Boost Python Performance: Parallelize Tasks with ThreadPool and map

Python’s reputation for parallel execution is often tarnished by confusing thread implementations and the Global Interpreter Lock (GIL). The author argues that misleading teaching materials, which focus on heavyweight class‑based producer/consumer patterns, are the main obstacle for everyday scripting.

Traditional examples

Typical tutorials showcase class and queue based multithreading code that resembles Java, requiring boilerplate worker classes, queues, and explicit join calls, which are error‑prone and over‑engineered for simple tasks.

import os
import PIL
from multiprocessing import Pool
from PIL import Image

SIZE = (75, 75)
SAVE_DIRECTORY = 'thumbs'

def get_image_paths(folder):
    return (os.path.join(folder, f) for f in os.listdir(folder) if 'jpeg' in f)

def create_thumbnail(filename):
    im = Image.open(filename)
    im.thumbnail(SIZE, Image.ANTIALIAS)
    base, fname = os.path.split(filename)
    save_path = os.path.join(base, SAVE_DIRECTORY, fname)
    im.save(save_path)

if __name__ == '__main__':
    folder = os.path.abspath('...')
    os.mkdir(os.path.join(folder, SAVE_DIRECTORY))
    images = get_image_paths(folder)
    pool = Pool()
    pool.map(creat_thumbnail, images)
    pool.close()
    pool.join()

The problem

This approach forces you to create template classes, manage queues, and write extensive synchronization code, which quickly becomes cumbersome as the number of workers grows.

Why not try map?

The map function, borrowed from functional languages like Lisp, provides a concise way to apply a function to every element of a sequence. In Python, both multiprocessing and its lightweight sibling multiprocessing.dummy expose a Pool.map method that can parallelize the operation.

Using multiprocessing.dummy gives you a thread‑based pool (useful for I/O‑bound work) while multiprocessing gives a process‑based pool (better for CPU‑bound work).

Hands‑on example

Import the pools:

from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool

Instantiate a thread pool (default size equals CPU count): pool = ThreadPool() Set a custom size when needed: pool = ThreadPool(4) # four worker threads Parallelize URL fetching with a single line: results = pool.map(urllib2.urlopen, urls) Benchmarking different pool sizes shows dramatic speedups:

# Single thread: 14.4 s
# 4‑thread pool: 3.1 s
# 8‑thread pool: 1.4 s
# 13‑thread pool: 1.3 s

Beyond URL fetching, the article presents a real‑world case: generating thumbnails for thousands of images (a CPU‑intensive task). The single‑process version takes ~27.9 s for 6 000 images, while a Pool.map version finishes in about 5.6 s.

pool = ThreadPool(4)
results = pool.map(urllib2.urlopen, urls)
pool.close()
pool.join()

These examples illustrate that replacing verbose loops and explicit thread management with map and a thread pool can reduce code to a few lines while delivering significant performance gains.

In summary, the article encourages Python developers to favor the simple map ‑based parallelism offered by multiprocessing and multiprocessing.dummy over heavyweight class‑based threading tutorials.