Using Python Queue and ThreadPool for Thread Synchronization and Parallel Task Execution

Python's queue module offers a built‑in, thread‑safe queue that eliminates the need for explicit locks when sharing data between threads, preventing deadlocks because the queue internally handles synchronization.

To use it, import with from queue import Queue. A simple producer‑consumer example creates a queue of size 3, starts two threads—one putting random numbers into the queue and the other retrieving and printing them—demonstrating the put/get workflow.

import time
from queue import Queue
from threading import Thread
from random import randint
my_queue = Queue(3)

def f1(my_queus):
    for i in range(3):
        time.sleep(1)
        num = randint(0,10)
        print(num)
        my_queue.put(num)

def f2(my_queus):
    for i in range(3):
        time.sleep(1)
        num = my_queue.get()
        print(num)

t1 = Thread(target=f1,args=(my_queue,))
t2 = Thread(target=f2,args=(my_queue,))
t1.start()
t2.start()
t1.join()
t2.join()

The program outputs pairs of numbers, showing that each item put into the queue is later retrieved, confirming the one‑in‑one‑out behavior.

When many threads are needed, a thread pool simplifies management. The multiprocessing.pool.ThreadPool class creates a fixed number of worker threads; tasks are submitted with apply_async, and the pool is closed and joined after all tasks are dispatched.

from multiprocessing.pool import ThreadPool
import time

def hello(name):
    print('hello,{}'.format(name))
    time.sleep(2)
    print('Bye')

t = ThreadPool(3)
for i in range(3):
    t.apply_async(hello, args=(i,))

t.close()
 t.join()

Running this code prints the greetings from the three threads almost simultaneously, followed by their goodbye messages, illustrating parallel execution.

ThreadPool also supports passing positional and keyword arguments to tasks. The following example creates a pool of two threads, defines two functions—one without arguments and one accepting *args and **kwargs —submits them, and prints status messages before and after closing the pool.

from multiprocessing.pool import ThreadPool
import time

pool = ThreadPool(2)

def task1():
    time.sleep(1)
    print("任务1完成")

def task2(*args, **kwargs):
    time.sleep(1)
    print("任务2完成:", args, kwargs)

pool.apply_async(task1)
pool.apply_async(task2, args=(1,2), kwds={'a':1,'b':2})
print("任务提交完成")
pool.close()
pool.join()
print("任务完成")

The output shows the submission message, the completion of each task with its arguments, and a final completion notice, confirming that both thread‑based and process‑based concurrency can boost performance in scenarios such as web crawling.