Master Python Multiprocessing and Coroutines: From Queues to Async Efficiency

Multiprocessing Overview

Although processes are independent and cannot directly access each other's data, Python provides inter‑process communication (IPC) mechanisms that allow data sharing and enable the use of multi‑core CPUs.

Why Not Always Use Threads

Threads consume significant resources when many are created and are best suited for I/O‑bound tasks; for CPU‑bound workloads, processes or coroutines are preferable.

Coroutines as a Lightweight Alternative

Coroutines (also called micro‑threads) are user‑level lightweight threads that can greatly improve efficiency, especially for I/O‑bound operations.

1. Multiprocessing Basics

Processes are managed by the operating system; Python can start a new process via its C interface, allowing full utilization of multiple CPU cores.

2. Creating Processes

3. Inter‑Process Communication

Since processes are isolated, a translation layer is required for them to exchange data. Common IPC methods include:

Queue

Queue communication works like the parent process giving a Queue to the child; the child puts serialized data into the Queue, which the parent later retrieves and deserializes. Because memory is separate, only serialized data can be transferred.

Pipe

Pipe works similarly to Queue, providing a bidirectional data channel between processes.

4. Process Locks

Even though memory is independent, concurrent printing can cause garbled output; a process lock prevents race conditions when multiple processes write to the console.

5. Process Pool

Creating a new process copies the parent’s memory; frequent creation can exhaust memory, so a process pool limits the number of concurrent processes.

6. Coroutine Overview

Coroutines, also known as micro‑threads, are lightweight user‑level threads. An analogy: while downloading a movie (I/O), you can simultaneously boil water and wash dishes (CPU‑light tasks), improving overall efficiency.

7. Creating and Using Coroutines with gevent

gevent is a third‑party library that enables concurrent asynchronous programming in Python.

Running a test shows that execution time is dominated by the longest sleep, demonstrating that coroutines can bypass blocking I/O and greatly improve efficiency.

IO operations (reading from disk, network, or memory) do not consume CPU, while computation does.

8. Simple Coroutine Web Crawling

9. Coroutine‑Based Socket Server

Using coroutines, a high‑performance socket server can be written; the underlying implementation uses multiple threads, but the coroutine approach yields higher efficiency and lower memory consumption.

10. Summary

Coroutines switch within a single thread, reducing resource consumption.

No need for atomic operations to control flow, simplifying the programming model.

High concurrency, high scalability, low cost.

Choosing the appropriate model—multiprocessing, multithreading, or coroutines—based on the specific scenario yields the most efficient solution.