Boost Image Preprocessing Speed in Python with Just 3 Lines of Code

By default, a Python program runs in a single process using one CPU core, which wastes the additional cores present in most modern hardware. The concurrent.futures module provides built‑in features that let you utilize all cores with only a few extra lines.

Standard method

The typical single‑core approach loops over files and processes each image sequentially:

import glob
import os
import cv2

# Loop through all jpg files in the current folder
# Resize each one to size 600x600
for image_filename in glob.glob("*.jpg"):
    # Read in the image data
    img = cv2.imread(image_filename)
    # Resize the image
    img = cv2.resize(img, (600, 600))

Running this script on a folder with 1,000 JPEG files took about 8 seconds on a six‑core i7‑8700K CPU.

Faster method

Using a process pool, the same task can be parallelized across all CPU cores with only three additional lines of code:

import glob
import os
import cv2
import concurrent.futures

def load_and_resize(image_filename):
    # Read in the image data
    img = cv2.imread(image_filename)
    # Resize the image
    img = cv2.resize(img, (600, 600))

# Create a pool of processes (one per CPU core by default)
with concurrent.futures.ProcessPoolExecutor() as executor:
    # Get a list of files to process
    image_files = glob.glob("*.jpg")
    # Distribute the work across the pool
    executor.map(load_and_resize, image_files)

Executing the parallel version reduced the runtime to roughly 1.14 seconds, a near‑six‑fold speedup.

Note that spawning multiple processes incurs overhead, so the speed gain may vary depending on the workload and hardware.

The parallel pool works best when the same operation is applied independently to many data items. It does not guarantee order, and it requires that the data be picklable. The following types are supported by Python’s multiprocessing pickling mechanism:

None, True, and False

Integers, floats, and complex numbers

Strings, bytes, and bytearrays

Tuples, lists, sets, and dictionaries containing only picklable objects

Functions defined at the top level of a module (not lambdas)

Built‑in functions defined at the top level of a module

Classes defined at the top level of a module

Instances of such classes whose __dict__ or __getstate__() result is picklable

If your processing requires a specific order of results or involves non‑picklable objects, this approach may not be suitable.