Unlock Python Multithreading: A Complete Guide to Threads, Locks, and Queues

1 Multithreading

1.1 Introduction

Multithreading is similar to running several programs simultaneously. Its advantages include moving long‑running tasks to the background, keeping the user interface responsive (e.g., showing a progress bar), potentially speeding up execution, and freeing resources during I/O‑bound operations such as user input, file I/O, or network communication.

Each thread has its own entry point, sequential execution flow, and exit point, but it cannot run independently; it must exist within an application that schedules its execution. A thread maintains a set of CPU registers called the thread context, which records the state of the instruction pointer and stack pointer—two crucial registers that identify the thread's position in the process address space.

Threads can be pre‑empted (interrupted) and may be temporarily suspended (sleep) while other threads run.

Threads are classified as: Kernel thread: created and terminated by the operating system kernel. User thread: implemented in user space without kernel support.

Python 3 provides two primary thread modules:

_thread

threading

(recommended)

The legacy thread module has been deprecated; use threading instead. For compatibility, Python 3 renamed thread to _thread.

1.2 Thread Modules

Python’s standard library offers _thread for low‑level thread creation and a simple lock, and threading for higher‑level abstractions. The threading module includes all _thread functions plus additional utilities: threading.current_thread(): returns the current thread object. threading.enumerate(): returns a list of all alive threads. threading.active_count(): returns the number of alive threads (same as len(threading.enumerate())). threading.Thread(target, args=(), kwargs={}, daemon=None): creates a Thread instance. target: function the thread will execute. args: positional arguments for the target as a tuple. kwargs: keyword arguments for the target as a dict. daemon: whether the thread is a daemon.

The threading.Thread class provides the following methods and attributes:

__init__(self, group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)

start(self)

: starts the thread and invokes its run() method. run(self): contains the code that the thread executes. join(self, timeout=None): blocks until the thread terminates or the optional timeout expires. is_alive(self): returns True if the thread is still running. getName(self) / setName(self, name): get or set the thread name. ident: unique identifier of the thread. daemon: flag indicating whether the thread is a daemon.

Simple thread example:

import threading
import time

def print_numbers():
    for i in range(5):
        time.sleep(1)
        print(i)

# Create thread
thread = threading.Thread(target=print_numbers)
# Start thread
thread.start()
# Wait for thread to finish
thread.join()

1.3 Creating Threads with _thread

Two ways to use threads in Python: function‑based or class‑based. The function approach calls _thread.start_new_thread():

_thread.start_new_thread(function, args[, kwargs])

Parameters: function: the thread function. args: a tuple of arguments passed to the function. kwargs: optional keyword arguments.

#!/usr/bin/python3
import _thread
import time

def print_time(threadName, delay):
    count = 0
    while count < 5:
        time.sleep(delay)
        count += 1
        print("%s: %s" % (threadName, time.ctime(time.time())))

try:
    _thread.start_new_thread(print_time, ("Thread-1", 2,))
    _thread.start_new_thread(print_time, ("Thread-2", 4,))
except:
    print("Error: unable to start thread")

while 1:
    pass

1.4 Creating Threads with threading

Subclass threading.Thread and override run() :

#!/usr/bin/python3
import threading
import time

exitFlag = 0

class myThread(threading.Thread):
    def __init__(self, threadID, name, delay):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.delay = delay
    def run(self):
        print("Start thread: " + self.name)
        print_time(self.name, self.delay, 5)
        print("Exit thread: " + self.name)

def print_time(threadName, delay, counter):
    while counter:
        if exitFlag:
            threadName.exit()
        time.sleep(delay)
        print("%s: %s" % (threadName, time.ctime(time.time())))
        counter -= 1

thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread(2, "Thread-2", 2)
thread1.start()
thread2.start()
thread1.join()
thread2.join()
print("Exit main thread")

1.5 Thread Synchronization Locks

When multiple threads modify shared data, race conditions can occur. Use Lock or RLock from the threading module. Both provide acquire() and release() methods. Example:

Consider a list of zeros. One thread sets each element to 1 from the end, while another thread reads and prints the list from the start. Without a lock, the output may contain a mixture of zeros and ones. Using a lock ensures the list is either all zeros or all ones when printed.

Always release a lock, preferably with try...finally to avoid deadlocks.

1.6 Thread‑Safe Queues (Queue)

Python’s queue module provides synchronized, thread‑safe queues: FIFO Queue , LIFO LifoQueue , and priority PriorityQueue . Common methods include: qsize(): size of the queue. empty(): True if empty. full(): True if full. get([block, timeout]): retrieve an item. get_nowait(): non‑blocking get. put(item): add an item. put_nowait(item): non‑blocking put. task_done(): signal completion of a task. join(): block until all items have been processed.

#!/usr/bin/python3
import queue
import threading
import time

exitFlag = 0

class myThread(threading.Thread):
    def __init__(self, threadID, name, q):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.q = q
    def run(self):
        print("Start thread: " + self.name)
        process_data(self.name, self.q)
        print("Exit thread: " + self.name)

def process_data(threadName, q):
    while not exitFlag:
        queueLock.acquire()
        if not workQueue.empty():
            data = q.get()
            queueLock.release()
            print("%s processing %s" % (threadName, data))
        else:
            queueLock.release()
        time.sleep(1)

threadList = ["Thread-1", "Thread-2", "Thread-3"]
nameList = ["One", "Two", "Three", "Four", "Five"]
queueLock = threading.Lock()
workQueue = queue.Queue(10)
threads = []
threadID = 1
for tName in threadList:
    thread = myThread(threadID, tName, workQueue)
    thread.start()
    threads.append(thread)
    threadID += 1

queueLock.acquire()
for word in nameList:
    workQueue.put(word)
queueLock.release()

while not workQueue.empty():
    pass

exitFlag = 1
for t in threads:
    t.join()
print("Exit main thread")

1.7 ThreadLocal

Each thread can have its own data using threading.local() . This avoids the need for explicit dictionaries and automatically provides thread‑local attributes.

import threading

# Global ThreadLocal object
local_school = threading.local()

def process_student():
    print('Hello, %s (in %s)' % (local_school.student, threading.current_thread().name))

def process_thread(name):
    local_school.student = name
    process_student()

t1 = threading.Thread(target=process_thread, args=('Alice',), name='Thread-A')
t2 = threading.Thread(target=process_thread, args=('Bob',), name='Thread-B')
t1.start()
t2.start()
t1.join()
t2.join()

The local_school object acts like a dict where each thread sees its own student attribute without affecting others. It is commonly used to store per‑thread database connections, HTTP sessions, or user identity information.

1.8 Thread Pool

For a higher‑level abstraction, Python’s multiprocessing.dummy.Pool (a thread‑based pool) can manage a pool of worker threads.

2 Multiprocessing vs. Multithreading

2.1 Differences

Multiprocessing

offers high stability because a crash in one child process does not affect others, though the master process can still be a single point of failure.

Creating processes is expensive, especially on Windows; the number of concurrent processes is limited by memory and CPU. Multithreading is generally faster but any thread crash can bring down the entire process because threads share memory.

On Windows, multithreading is more efficient than multiprocessing; on Unix, both models are used, sometimes combined.

3.2 Thread Switching

When many threads or processes run, the operating system spends time saving the current execution context and loading the next one. Excessive switching leads to high overhead and reduced overall performance.

3.3 CPU‑Bound vs. I/O‑Bound

CPU‑bound tasks perform intensive calculations and benefit from a number of threads equal to the number of CPU cores. They are best written in compiled languages like C. I/O‑bound tasks spend most of their time waiting for network or disk operations; they can use many more threads (e.g., 2×CPU cores) and are well suited to high‑level languages like Python.

3.4 Asynchronous I/O

Modern operating systems support asynchronous I/O, allowing a single‑process, single‑thread program to handle many concurrent I/O operations via an event‑driven model (e.g., Nginx). In Python, this model is expressed with coroutines and the asyncio library.