Boost Python Performance: Master Multiprocessing with Real Code Examples

Python multiprocessing enables parallel execution across multiple CPU cores, unlike multithreading which is limited by the Global Interpreter Lock (GIL). Each process runs its own interpreter, making it ideal for CPU‑intensive workloads.

To use multiprocessing, import the built‑in multiprocessing module. Below is a simple example that creates a single child process:

import multiprocessing

def worker():
    """Child process work function"""
    print("Starting worker")
    # place time‑consuming tasks here
    print("Finished worker")

if __name__ == '__main__':
    p = multiprocessing.Process(target=worker)  # create a child process
    p.start()                                 # start the child process
    p.join()                                  # wait for it to finish
    print("Parent process finished")

In this example, worker() is the function executed by the child process. The Process class creates the process, start() launches it, and join() waits for completion.

For handling multiple tasks concurrently, the Pool class can manage a pool of worker processes. The following code creates a pool of four processes and runs the worker() function in parallel using map():

import multiprocessing

def worker(num):
    """Child process work function"""
    print(f"Starting worker {num}")
    # place time‑consuming tasks here
    print(f"Finished worker {num}")

if __name__ == '__main__':
    with multiprocessing.Pool(processes=4) as pool:
        pool.map(worker, [1, 2, 3, 4])
    print("Parent process finished")

The Pool distributes the arguments to worker() across the four processes, and map() collects the results.

Another common pattern uses imap_unordered() for asynchronous execution. The example below computes squares of numbers with a simulated delay:

import multiprocessing
import time

def square(x):
    """Compute square"""
    time.sleep(1)  # simulate a time‑consuming calculation
    return x * x

if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        for result in pool.imap_unordered(square, range(10)):
            print(result)

imap_unordered()

returns results as soon as each task finishes, so the output order may differ from the input order.

Common scenarios for Python multiprocessing include:

Data processing – split large datasets into chunks and process them in parallel.

Web crawling – assign different URLs to separate processes for concurrent fetching.

Image processing – apply transformations to many images simultaneously.

Key considerations when using multiprocessing:

Inter‑process communication – processes have separate memory, so use pipes, queues, or other mechanisms provided by multiprocessing for data exchange.

Process pool sizing – limit the number of concurrent processes to avoid exhausting system resources.

Memory usage – each process consumes its own memory; creating too many can lead to crashes.

In summary, Python's multiprocessing module and the Pool class make it straightforward to create and manage multiple processes, enabling efficient parallel execution of CPU‑bound tasks.