Python Concurrency Programming: Threads, Processes, GIL, and Thread Pools

Introduction

Python introduces concurrent programming to improve execution speed. Mastering concurrency—especially multithreading and multiprocessing—is essential for senior developers because they address I/O‑bound and CPU‑bound problems respectively.

Why introduce concurrency?

Examples show that a sequential web crawler takes 1 hour, while concurrent downloading reduces it to 20 minutes; a personalized homework recommendation system processes 3,000 users in 17 s with a single thread but only 0.5 s with multiprocessing. The conclusions are:

Concurrency speeds up programs by better CPU utilization.

It enables task decomposition and parallel execution.

It is a required skill for advanced programmers.

Methods to accelerate Python programs

Python supports several long‑term concurrency techniques: multithreading, multiprocessing, and generator‑based coroutines (the latter not covered here). An illustration of these methods is provided.

Choosing between multithreading and multiprocessing

CPU‑bound tasks benefit from multiprocessing, while I/O‑bound tasks benefit from multithreading. The article explains the characteristics of each:

CPU‑bound (high CPU usage): examples include compression, encryption, regex search.

I/O‑bound (low CPU usage): examples include file handling, web crawling, database access.

Python’s concurrency primitives

Key modules and classes:

threading – lightweight threads sharing memory.

multiprocessing – separate processes that can run on multiple CPU cores.

asyncio – asynchronous I/O in a single thread.

Synchronization tools: Lock , Queue , thread pools, process pools.

Global Interpreter Lock (GIL)

The GIL ensures that only one Python thread executes bytecode at a time, which limits multithreaded CPU‑bound performance. While Python can create many threads, they contend for the GIL, so only one runs on the CPU at any moment.

Reasons for the GIL include simplifying memory management and reference counting. Removing the GIL is historically difficult.

Mitigating GIL limitations

For I/O‑bound workloads, multithreading remains effective because threads spend most of their time waiting. For CPU‑bound workloads, use multiprocessing to achieve true parallelism across cores.

Multithreading practice

Example code shows how to create and start a thread:

# 1. Prepare a function
def my_func(a, b):
    craw(a, b)

# 2. Create a thread
t = threading.Thread(target=my_func, args=(100, 200))

# 3. Start the thread
t.start()

# 4. Wait for completion
t.join()

A complete crawler example compares single‑threaded and multithreaded execution, demonstrating a speedup from ~3.8 s to 0.37 s.

Thread safety

When multiple threads modify shared variables, race conditions occur. The article shows a counter example where three threads increment a global variable 100 000 times each, producing an incorrect final value due to lack of synchronization.

Locks can resolve this:

# try‑finally style
import threading
lock = threading.Lock()
lock.acquire()
try:
    # critical section
    pass
finally:
    lock.release()

# with‑statement style
import threading
lock = threading.Lock()
with lock:
    # critical section
    pass

Thread pools

Thread pools reuse worker threads to avoid the overhead of creating and destroying threads. The pool workflow includes checking for idle threads, creating new ones up to corePoolSize , queuing tasks, and expanding up to maximumPoolSize when needed.

Benefits:

Performance gain by reusing threads.

Suitable for bursty workloads with many short‑lived tasks.

Prevents system overload from excessive thread creation.

Simpler syntax compared to manual thread management.

Example using ThreadPoolExecutor :

from concurrent.futures import ThreadPoolExecutor, as_completed

# map style (order preserved)
with ThreadPoolExecutor() as pool:
    results = pool.map(craw, urls)
    for result in results:
        print(result)

# future style (order not guaranteed)
with ThreadPoolExecutor() as pool:
    futures = [pool.submit(craw, url) for url in urls]
    for future in as_completed(futures):
        print(future.result())

Multiprocessing

Multiprocessing overcomes the GIL by running separate processes on multiple CPUs. It is advantageous for CPU‑bound tasks, whereas multithreading remains useful for I/O‑bound workloads.

Performance comparison in a personalized homework recommendation scenario shows that multiprocessing only outperforms single‑threaded execution when the sample size exceeds about 70 items.

References

1. https://blog.csdn.net/liuxiao723846/article/details/108026782 2. http://c.biancheng.net/view/2606.html 3. https://blog.csdn.net/gua_niu123/article/details/111350343 4. https://blog.csdn.net/qq_50840738/article/details/123861602 5. https://blog.51cto.com/u_14575624/4369560 6. 蚂蚁学Python – Python concurrency 7. 《Python+Cookbook》第三版中文v3.0.0