Master Python Multithreading: Processes, GIL, and Threading Modules Explained

Introduction & Motivation

Consider processing 10,000 items where each item takes 1 s to compute and 0.1 s to read. Using a single execution sequence would take 11,000 s, which is too long. Multithreading allows overlapping I/O and computation to reduce total time.

Processes and Threads

What is a Process?

A process (heavyweight) is an instance of a program with its own address space, memory, and data stack. The OS schedules processes and allocates CPU time fairly. Processes communicate via inter‑process communication (IPC) but cannot share memory directly.

What is a Thread?

A thread (lightweight) runs within a process, sharing the same memory space. Threads can be seen as mini‑processes that execute concurrently inside the main process.

Thread states are illustrated below.

Threads have a start, run, and finish phase, a program counter, and can be pre‑empted or put to sleep, allowing other threads to run. Shared memory makes communication easy but can cause race conditions.

Global Interpreter Lock (GIL)

The GIL is not a Python language feature but an implementation detail of CPython. It is a mutex that prevents multiple native threads from executing Python bytecode simultaneously because CPython’s memory management is not thread‑safe.

In multithreaded programs the GIL is released before I/O calls, allowing other threads to run while one thread waits for I/O. Consequently, I/O‑bound Python programs benefit from multithreading, whereas CPU‑bound programs do not.

Exiting Threads

Threads terminate by returning from their target function or by calling thread.exit(), sys.exit(), or raising SystemExit. Threads cannot be killed directly.

Threading in Python

Python supports POSIX‑compatible threads on most platforms. The threading module provides the high‑level Thread class and many synchronization primitives.

thread vs threading modules

The low‑level thread module offers basic thread and lock support; the higher‑level threading module adds richer functionality and is recommended.

Avoid thread because it lacks many synchronization primitives and can cause unexpected termination of threads when the main thread exits.

threading primitives

Lock : basic mutex.

RLock : re‑entrant lock allowing the same thread to acquire multiple times.

Event : flag for signaling between threads.

Condition : combines a lock with a wait/notify mechanism.

Semaphore / BoundedSemaphore : counting semaphore.

Timer : runs a function after a delay.

Thread class methods

run()

start()

join([timeout])

is_alive()

name

daemon

Creating Threads

Three common patterns:

Instantiate Thread with a target function.

Instantiate Thread with a callable object.

Subclass Thread and override run().

Queue Module and Producer‑Consumer

The queue module provides a thread‑safe FIFO queue for producer‑consumer patterns. Key methods include put(), get(), qsize(), empty(), and full().

FAQ

1. Difference between processes and threads – Processes have separate memory; threads share memory.

2. Which Python programs benefit from multithreading? – I/O‑bound programs benefit because the GIL is released during I/O.

3. Multithreading on multi‑core CPUs – The GIL limits parallelism for CPU‑bound work; only one thread runs Python bytecode at a time.

4. Thread pool for producer‑consumer – Use a pool of consumer threads to handle multiple items concurrently.