Deep Dive into Python Concurrency: Threads, Processes, and Coroutines (Part 1)

1. Three Major Python Concurrency Paradigms

Threading

Threading is lightweight but constrained by the Global Interpreter Lock (GIL). Python threads wrap native OS threads and are scheduled by the OS. Threads share the same memory space, resulting in low communication cost.

import threading
import time

def worker(num):
    """Simulate an I/O‑bound task"""
    print(f'线程{num}开始执行')
    time.sleep(2)  # simulate I/O
    print(f'线程{num}执行完成')

threads = []
for i in range(5):
    t = threading.Thread(target=worker, args=(i,))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

Performance characteristics: low creation cost, small context‑switch overhead, shared memory, but requires synchronization and cannot achieve true parallelism for CPU‑bound work because of the GIL.

Typical scenarios: I/O‑intensive tasks (network requests, file I/O), background work in GUI applications, and situations needing quick response with modest computation.

Multiprocessing

Multiprocessing provides true parallelism by spawning separate Python interpreter instances, each with its own memory space, thus bypassing the GIL.

from multiprocessing import Pool, cpu_count
import time

def cpu_intensive_task(n):
    """CPU‑bound task example"""
    result = 0
    for i in range(n):
        result += i * i
    return result

if __name__ == '__main__':
    processes = cpu_count()  # use all CPU cores
    with Pool(processes) as pool:
        start_time = time.time()
        results = pool.map(cpu_intensive_task, [10_000_000] * processes)
        elapsed = time.time() - start_time
        print(f'使用{processes}个进程，耗时：{elapsed:.2f}秒')

Performance characteristics: memory isolation, no shared state, inter‑process communication (IPC) required, higher creation and context‑switch cost, but fully utilizes multi‑core CPUs.

Typical scenarios: CPU‑intensive computations (data analysis, scientific computing), services that need process isolation, and heavy, independent tasks.

Coroutines (asyncio)

Coroutines offer a lightweight, user‑space concurrency model driven by an event loop. They run in a single OS thread and voluntarily yield control during I/O operations.

import asyncio
import aiohttp

async def fetch_url(session, url):
    """Asynchronously fetch a web page"""
    async with session.get(url) as response:
        return await response.text()

async def main():
    urls = [
        'https://httpbin.org/delay/1',
        'https://httpbin.org/delay/2',
        'https://httpbin.org/delay/1',
    ]
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_url(session, url) for url in urls]
        results = await asyncio.gather(*tasks)
        print(f'获取了{len(results)}个页面')

# Python 3.7+
asyncio.run(main())

Performance characteristics: extremely lightweight, minimal context‑switch overhead, can handle tens of thousands of concurrent connections on a single machine, but requires asynchronous programming mindset and special syntax.

Typical scenarios: high‑concurrency I/O‑bound services, microservice gateways, API proxies, and real‑time message‑push systems.

2. In‑Depth GIL Analysis: Mechanism, Impact, and Strategies

How the GIL Works

The Global Interpreter Lock (GIL) is a mutex in the CPython interpreter that ensures only one thread executes Python bytecode at a time.

# Simple demonstration of GIL impact
import threading
import time

counter = 0

def increment():
    global counter
    for _ in range(1000000):
        counter += 1  # not an atomic operation

# Create two threads that increment the counter simultaneously
t1 = threading.Thread(target=increment)
t2 = threading.Thread(target=increment)

t1.start()
t2.start()

t1.join()
t2.join()
print(f'预期值: 2000000, 实际值: {counter}')
# Due to the GIL and thread switching, the actual result is usually less than expected

The GIL is released in three situations: during I/O operations (file read/write, network requests), during blocking calls such as time.sleep(), and after executing a certain number of bytecode instructions (configurable via sys.setcheckinterval).

Actual Impact of the GIL on Web Applications

For CPU‑bound workloads, the GIL prevents multithreaded code from gaining performance; for I/O‑bound workloads, the impact is limited because threads release the GIL while waiting for I/O. Mixed workloads exhibit clear bottlenecks.

Mitigation Strategy for Web Applications

# Hybrid web server example mixing processes and coroutines
from concurrent.futures import ProcessPoolExecutor
import asyncio
import aiohttp
from aiohttp import web

# CPU‑bound tasks run in a process pool
executor = ProcessPoolExecutor(max_workers=4)

async def cpu_bound_handler(request):
    """Handle a CPU‑intensive request"""
    data = await request.json()
    loop = asyncio.get_event_loop()
    result = await loop.run_in_executor(
        executor,
        heavy_computation,
        data,
    )
    return web.json_response({'result': result})

async def io_bound_handler(request):
    """Handle an I/O‑intensive request"""
    async with aiohttp.ClientSession() as session:
        async with session.get('https://api.example.com/data') as resp:
            data = await resp.json()
            return web.json_response(data)

def heavy_computation(data):
    """CPU‑intensive computation function"""
    import math
    return sum(math.sqrt(i) for i in range(data['n']))

app = web.Application()
app.router.add_post('/compute', cpu_bound_handler)
app.router.add_get('/fetch', io_bound_handler)

web.run_app(app)

By delegating CPU‑heavy work to separate processes and handling I/O with asynchronous coroutines, a web service can mitigate GIL limitations and achieve higher overall throughput.

To be continued…