Boost Python Performance: Simple Parallelism with map and ThreadPool

Why Python Parallelism Is Misunderstood

Python’s reputation for poor parallelism is often blamed on technical issues like the GIL, but the real problem is misleading teaching that focuses on heavyweight thread‑pool examples without addressing everyday scripting needs.

Traditional Examples

Typical tutorials show class‑based producer/consumer patterns that resemble Java code and require boilerplate queues and explicit thread management.

import os
import PIL

from multiprocessing import Pool
from PIL import Image

SIZE = (75,75)
SAVE_DIRECTORY = 'thumbs'

def get_image_paths(folder):
    return (os.path.join(folder, f)
            for f in os.listdir(folder)
            if 'jpeg' in f)

def create_thumbnail(filename):
    im = Image.open(filename)
    im.thumbnail(SIZE, Image.ANTIALIAS)
    base, fname = os.path.split(filename)
    save_path = os.path.join(base, SAVE_DIRECTORY, fname)
    im.save(save_path)

if __name__ == '__main__':
    folder = os.path.abspath('11_18_2013_R000_IQM_Big_Sur_Mon__e10d1958e7b766c3e840')
    os.mkdir(os.path.join(folder, SAVE_DIRECTORY))
    images = get_image_paths(folder)
    pool = Pool()
    pool.map(create_thumbnail, images)
    pool.close()
    pool.join()

These examples are verbose and error‑prone for simple scripts.

The Problem with Boilerplate

Requires a template class.

Needs a queue to pass objects.

Often demands extra methods for bidirectional communication.

Why Not Use map?

The built‑in map function, originating from functional languages, can replace lengthy thread‑pool code by applying a function to each element of a sequence, handling iteration, argument passing, and result collection automatically.

urls = ['http://www.yahoo.com', 'http://www.reddit.com']
results = map(urllib2.urlopen, urls)

Internally this is equivalent to a simple loop, but when combined with the proper library it executes in parallel.

Python provides two libraries that implement map with parallelism: multiprocessing (process‑based) and its lesser‑known sibling multiprocessing.dummy (thread‑based).

Hands‑On Example

Import the pools:

from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool

Instantiate a thread pool (default size equals CPU count):

pool = ThreadPool()

For I/O‑bound work you may set a custom size, e.g., four workers:

pool = ThreadPool(4)  # Sets the pool size to 4

Use pool.map to fetch URLs in parallel:

import urllib2
urls = [
    'http://www.python.org',
    'http://www.python.org/about/',
    # ... more URLs ...
]
pool = ThreadPool(4)
results = pool.map(urllib2.urlopen, urls)
pool.close()
pool.join()

On the author’s machine a single‑threaded run took 14.4 seconds, while a 4‑worker pool reduced it to 3.1 seconds, an 8‑worker pool to 1.4 seconds, and a 13‑worker pool to 1.3 seconds, showing diminishing returns beyond nine workers.

Real‑World CPU‑Intensive Example

Generating thumbnails for thousands of images benefits from process‑based parallelism:

from multiprocessing import Pool
# same thumbnail functions as above
pool = Pool()
pool.map(create_thumbnail, images)
pool.close()
pool.join()

Replacing a manual loop with pool.map cut processing time from 27.9 seconds to about 5.6 seconds for 6 000 images.

Original article: http://dwz.date/bZGa