Python Multiprocessing: Using the multiprocessing Module for Parallel Execution

Python multiprocessing (Multiprocessing) is a technique that utilizes multiple CPU cores for parallel processing, differing from multithreading which is limited by the Global Interpreter Lock (GIL). Each process runs its own interpreter, making it more effective for CPU‑intensive tasks.

To use multiprocessing in Python, import the built‑in multiprocessing module, which provides classes and functions for creating and managing processes. Below is a simple example that creates a single child process.

<code>import multiprocessing

def worker():
    """Child process work function"""
    print("Starting worker")
    # place time‑consuming tasks here
    print("Finished worker")

if __name__ == '__main__':
    # create a child process
    p = multiprocessing.Process(target=worker)
    # start the child process
    p.start()
    # wait for the child process to finish
    p.join()
    print("Parent process finished")
</code>

In this example, the worker() function defines the task for the child process. The multiprocessing.Process class creates the process, start() launches it, and join() waits for its completion, after which the parent prints a final message.

Beyond a single process, the Pool class can create a pool of worker processes to handle multiple tasks concurrently. The following example creates a pool of four processes and uses map() to run worker() in parallel.

<code>import multiprocessing

def worker(num):
    """Child process work function"""
    print(f"Starting worker {num}")
    # place time‑consuming tasks here
    print(f"Finished worker {num}")

if __name__ == '__main__':
    # create a pool with 4 processes
    with multiprocessing.Pool(processes=4) as pool:
        # execute worker in parallel using map
        pool.map(worker, [1, 2, 3, 4])
    print("Parent process finished")
</code>

This demonstrates how Pool distributes the list of arguments to the worker() function, allowing the four processes to run simultaneously.

Common scenarios for Python multiprocessing include data processing, web crawling, and image processing—any workload that can be divided into independent tasks and benefits from parallel execution.

When using multiprocessing, be aware of several considerations:

Inter‑process communication (IPC): processes have separate memory spaces, so data must be shared via pipes, queues, or other mechanisms provided by the multiprocessing module.

Process pool size: controlling the number of concurrent processes prevents excessive resource consumption.

Memory limits: creating too many processes can exhaust system memory and cause crashes.

Another example shows asynchronous execution with imap_unordered() , which returns results as soon as each task finishes, regardless of order.

<code>import multiprocessing
import time

def square(x):
    """Compute square"""
    time.sleep(1)  # simulate a time‑consuming calculation
    return x * x

if __name__ == '__main__':
    # create a process pool
    with multiprocessing.Pool() as pool:
        # asynchronously execute tasks
        for result in pool.imap_unordered(square, range(10)):
            print(result)
</code>

The imap_unordered() method yields results as they become available, which can improve throughput when task durations vary.

Other multiprocessing methods include imap() (preserves order), apply() , and apply_async() , which can be used for single or asynchronous task execution.

In summary, Python's multiprocessing module and the Pool class provide powerful tools for efficiently handling CPU‑bound workloads by creating and managing multiple processes that run tasks in parallel.