Accelerating Image Pre‑processing in Python with a Three‑Line Multiprocessing Trick

Python is the preferred language for machine‑learning workflows, but when processing large image datasets the default single‑process execution wastes CPU cores, leading to poor performance.

The typical approach iterates over files, reads each image, and resizes it using a simple for loop, as shown in the standard script:

<code>import glob
import os
import cv2

# Loop through all jpg files in the current folder
# Resize each one to size 600x600
for image_filename in glob.glob("*.jpg"):
    # Read in the image data
    img = cv2.imread(image_filename)
    # Resize the image
    img = cv2.resize(img, (600, 600))
</code>

Running this script on a folder with 1,000 JPEGs on an Intel i7‑8700K (6 cores) took about 8 seconds, which is far from optimal.

By leveraging Python's concurrent.futures module, the same task can be parallelized across all CPU cores with only three additional lines of code:

<code>import glob
import os
import cv2
import concurrent.futures

def load_and_resize(image_filename):
    img = cv2.imread(image_filename)
    img = cv2.resize(img, (600, 600))
    return img

with concurrent.futures.ProcessPoolExecutor() as executor:
    image_files = glob.glob("*.jpg")
    executor.map(load_and_resize, image_files)
</code>

This method splits the file list among multiple processes, allowing each core to work on a subset simultaneously. On the same hardware the execution time dropped to roughly 1.14 seconds, a near‑six‑fold speedup.

While parallel pools provide substantial gains, they also introduce overhead for process creation and inter‑process communication, so the improvement may vary with workload size and system configuration.

The technique works best for embarrassingly parallel tasks where each data item can be processed independently; it is unsuitable when the order of results matters or when the data cannot be pickled for multiprocessing.

Python can parallelize any picklable object, including basic types (ints, floats, strings), collections (lists, tuples, sets, dicts), and top‑level functions or classes defined in modules.