Unlock Full CPU Power in Python: A Hands‑On Guide to Multiprocessing

Python does not always fully utilize all CPU cores because of the Global Interpreter Lock (GIL), which allows only one thread to execute Python bytecode at a time. While the GIL simplifies many aspects of Python programming, it can become a bottleneck for CPU‑bound tasks.

The solution is the multiprocessing module. In this article we explore the basics of Python multiprocessing and see how it can help you harness the full potential of your machine’s CPU cores.

When to use multiprocessing: CPU‑intensive tasks such as numerical computation, image processing, or data transformation where the CPU is the bottleneck.

Parallelism: Multiprocessing lets you distribute work across multiple processes, achieving true parallel execution.

Getting Started with Multiprocessing

We begin with a simple example that computes the squares of a list of numbers.

Example: Compute Squares

<code>import multiprocessing
import time

def square(n):
    """Compute the square of a number"""
    time.sleep(1)  # simulate a time‑consuming task
    return n * n

if __name__ == "__main__":
    numbers = [1, 2, 3, 4, 5]
    # Sequential execution
    start_time = time.time()
    results = [square(n) for n in numbers]
    print(f"Sequential results: {results}")
    print(f"Sequential time: {time.time() - start_time:.2f} seconds")
    # Parallel execution
    start_time = time.time()
    with multiprocessing.Pool(processes=5) as pool:
        results = pool.map(square, numbers)
    print(f"Parallel results: {results}")
    print(f"Parallel time: {time.time() - start_time:.2f} seconds")
</code>

Output:

<code>Sequential results: [1, 4, 9, 16, 25]
Sequential time: 5.00 seconds
Parallel results: [1, 4, 9, 16, 25]
Parallel time: 1.00 seconds
</code>

In parallel execution, the Pool object manages multiple processes and distributes the tasks, dramatically reducing total execution time.

Key Multiprocessing Concepts

Process : Each process runs independently with its own memory space. You can create a process with the Process class: <code>from multiprocessing import Process def greet(name): print(f"Hello, {name}!") if __name__ == "__main__": p = Process(target=greet, args=("Alice",)) p.start() p.join() </code>

Process Pool : The Pool class simplifies parallel processing by managing a pool of worker processes. Example: Using apply and apply_async <code>from multiprocessing import Pool import time def cube(n): return n ** 3 if __name__ == "__main__": with Pool(processes=2) as pool: result = pool.apply(cube, (3,)) print(f"apply result: {result}") async_result = pool.apply_async(cube, (4,)) print(f"apply_async result: {async_result.get()}") </code> Output: <code>apply result: 27 apply_async result: 64 </code> map : Distribute a function over multiple inputs. apply : Execute a function with a set of arguments. apply_async : Execute a function asynchronously.

Inter‑Process Synchronization and Communication

When working with multiple processes you may need to share data or coordinate actions. The multiprocessing module provides several tools:

Queue : Safe communication between processes. <code>from multiprocessing import Process, Queue def producer(queue): for i in range(5): queue.put(i) def consumer(queue): while not queue.empty(): print(queue.get()) if __name__ == "__main__": queue = Queue() p1 = Process(target=producer, args=(queue,)) p2 = Process(target=consumer, args=(queue,)) p1.start(); p1.join() p2.start(); p2.join() </code> Output: <code>0 1 2 3 4 </code>

Lock : Prevent concurrent access to shared resources. <code>from multiprocessing import Process, Lock def task(lock, i): with lock: print(f"Process {i} is working") if __name__ == "__main__": lock = Lock() processes = [Process(target=task, args=(lock, i)) for i in range(5)] for p in processes: p.start() for p in processes: p.join() </code> Output: <code>Process 0 is working Process 1 is working Process 2 is working Process 3 is working Process 4 is working </code>

Real‑World Example: Image Processing

Suppose you have a folder with hundreds of images that need to be resized. Processing them sequentially is slow, but using multiprocessing you can handle several images simultaneously.

<code>from multiprocessing import Pool
from PIL import Image
import os

def resize_image(image_path):
    """Resize an image to 100x100 pixels"""
    with Image.open(image_path) as img:
        img = img.resize((100, 100))
        img.save(f"resized_{os.path.basename(image_path)}")

if __name__ == "__main__":
    image_folder = "path_to_images"
    image_files = [os.path.join(image_folder, f) for f in os.listdir(image_folder) if f.endswith('.jpg')]
    with Pool(processes=4) as pool:
        pool.map(resize_image, image_files)
    print("Image resizing completed!")
</code>

Output (illustrative):

<code>resized_image1.jpg
resized_image2.jpg
resized_image3.jpg
... (all images in the folder are resized)
Image resizing completed!
</code>

Best Practices

Avoid shared state : Each process has its own memory; minimize reliance on shared data to prevent race conditions.

Use if __name__ == "__main__" to protect the entry point and avoid recursive process creation.

Limit the number of processes : Do not create more processes than available CPU cores.

Conclusion

The multiprocessing module is a powerful tool for parallelizing CPU‑bound tasks in Python. By mastering its fundamentals, you can fully leverage all CPU cores and achieve significant performance gains.

Stay tuned for more deep dives into multiprocessing concepts.