Master Python ThreadPoolExecutor and ProcessPoolExecutor for Efficient Concurrency

Preface

Hello, I'm Xingqi Ba. We all know that Java, C++, Go, and Python all have the concept of threads.

However, threads cannot be created arbitrarily because each thread has a cost, similar to hiring employees.

Therefore we usually introduce the concept of a thread pool.

A thread pool essentially hires a fixed number of workers and assigns tasks to them; when a worker finishes a task it fetches a new one, preventing system overload.

Fun Moment

Ideal Multithreading

Actual Multithreading

concurrent.futures import …

Since Python 3.2, the concurrent.futures module is built‑in, providing two main executors: ThreadPoolExecutor (thread pool) and ProcessPoolExecutor (process pool). There is no separate “async pool”.

Personal view: Asynchronous programming can be fast, but outside of Go the implementations are often awkward.

Thread Pool

Example code

import time
from concurrent.futures import ThreadPoolExecutor
import random

# max_workers indicates the number of workers in the pool
pool = ThreadPoolExecutor(max_workers=10)

# task list
task_list = ["任务1", "任务2", "任务3", "任务4"]

def handler(task_name):
    # random sleep to simulate variable execution time
    n = random.randrange(5)
    time.sleep(n)
    print(f"任务内容:{task_name}")

if __name__ == '__main__':
    for task in task_list:
        """ submit sends all tasks to the pool without blocking;
            each thread picks up a task; if there are more tasks than threads,
            completed threads take new tasks. """
        pool.submit(handler, task)
    print("main执行完毕")

Execution result

Problem discovered

The main thread finishes before child threads; we need the main thread to wait for all workers.

Calling pool.shutdown() makes the main thread wait until all worker threads complete.

pool.shutdown()

Updated example:

if __name__ == '__main__':
    for task in task_list:
        pool.submit(handler, task)
    pool.shutdown()
    print("main执行完毕")

Callback with add_done_callback

add_done_callback

registers a function to be called when a future completes, allowing access to the result.

import time
from concurrent.futures import ThreadPoolExecutor
import random
from concurrent.futures._base import Future

pool = ThreadPoolExecutor(max_workers=10)
task_list = ["任务1", "任务2", "任务3", "任务4"]

def handler(task_name):
    n = random.randrange(5)
    time.sleep(n)
    print(f"任务内容:{task_name}")
    return f"任务内容:{task_name}"

def done(res: Future):
    print("done拿到的返回值:", res.result())

if __name__ == '__main__':
    for task in task_list:
        future = pool.submit(handler, task)
        future.add_done_callback(done)
    pool.shutdown()
    print("main执行完毕")

Multiprocessing

Because of the Global Interpreter Lock (GIL), Python threads cannot utilize multiple CPU cores, making them suitable for I/O‑bound tasks. For CPU‑bound workloads such as image compression or video streaming, use a process pool.

Switching to ProcessPoolExecutor only requires changing the import and the executor creation:

from concurrent.futures import ProcessPoolExecutor
...
pool = ProcessPoolExecutor(max_workers=10)
...

Summary

This article covered Python’s built‑in thread pool and process pool via ThreadPoolExecutor and ProcessPoolExecutor. Both share the same interface: pool.submit(), pool.shutdown(), and future.add_done_callback(). These tools are sufficient for most concurrency needs in Python.