From Threads to Coroutines: Scaling to Millions of QPS

Introduction

When a long‑connection push gateway needed to keep millions of devices online, the one‑request‑one‑thread model quickly exhausted memory despite low CPU usage, because each thread reserved a megabyte‑scale stack and incurred kernel‑mode context switches.

Why Threads Are Expensive

Memory cost : Linux gives each thread an 8 MB virtual stack; even conservatively counting 1 MB per thread, 100 k threads already consume ~100 GB, plus thread control blocks.

Scheduling cost : Threads are kernel‑scheduled; creation, destruction and context switches involve saving registers, flushing caches and updating kernel data structures, typically costing 1–several microseconds. With tens of thousands of ready threads, the scheduler becomes a bottleneck.

Cache cost : Frequent thread switches pollute CPU caches, forcing the working set to be reloaded from memory, which degrades throughput.

Combined, these three costs cause the classic C10K problem to become a C10M problem when aiming for millions of connections and tens of millions of QPS.

Why One‑Thread‑Per‑Request Fails

In I/O‑bound services a thread spends most of its lifetime waiting for database, RPC or cache responses, holding its stack and kernel slot while doing nothing. Adding more threads only linearly increases memory and scheduling overhead, eventually nullifying any throughput gains.

What a Coroutine Is

A coroutine is a lightweight execution flow scheduled in user space. It differs from a thread in two ways: the scheduler runs in user space, and the coroutine yields cooperatively rather than being pre‑empted by the kernel.

Because switching happens in user space and each coroutine’s stack can be as small as a few kilobytes, a 1 GB machine can host hundreds of thousands of coroutines versus only a thousand threads.

Stackful vs. Stackless Coroutines

Stackful (e.g., Go goroutine) : Each coroutine has its own stack, which can grow dynamically. This makes the programming model transparent—any function can yield at any depth.

Stackless (e.g., Kotlin, Rust async, Python asyncio) : The compiler transforms coroutine functions into state machines; local variables are lifted to heap‑allocated structs. This reduces memory usage but requires explicit async / await and propagates “async‑ness” up the call chain.

M:N Scheduling – Go’s GMP Model

Go uses a three‑entity model: G (goroutine), M (machine, i.e., OS thread), and P (processor). The number of Ps defaults to the CPU core count. Goroutines (G) are scheduled onto a small set of OS threads (M) via local queues on each P, avoiding global locks and enabling work‑stealing.

The netpoller registers I/O events and wakes the corresponding goroutine without blocking the underlying thread, turning blocking I/O into coroutine suspension.

Synchronous Code, Asynchronous Performance

Coroutines let developers write sequential, blocking‑style code while the runtime performs non‑blocking I/O under the hood, eliminating callback hell and preserving readability.

Language Ecosystem Overview

Go: stackful goroutine, CSP channels, first‑class support.

Kotlin: stackless coroutines with suspend, structured concurrency.

Java: Project Loom virtual threads (stackful) that keep the traditional Thread API.

Python: asyncio (stackless) with event loop.

C++20: language‑level coroutines, usually used via libraries.

Rust: stackless async / await with runtimes like Tokio.

Java Virtual Threads

Virtual threads are lightweight JVM‑managed threads that run on a small pool of platform threads. They retain the standard Thread API, so existing blocking code can be run with minimal changes, achieving massive concurrency without rewriting code.

However, virtual threads can be “pinned” when blocking inside synchronized sections, losing their advantage.

CSP Model – Communicating Sequential Processes

Instead of sharing memory, coroutines communicate via channels, transferring ownership of data and reducing lock contention. While not a silver bullet, CSP simplifies reasoning about concurrency in highly concurrent systems.

Pitfalls of Coroutines

Cooperative scheduling fails for CPU‑bound tasks lacking yield points, causing starvation.

Debugging is harder because stack traces may be fragmented across yields.

Stackless coroutines suffer from async‑propagation (“function staining”).

Real‑World Migration Example

Replacing one‑thread‑per‑request with coroutines on the same hardware increased concurrent connections from 80 k to several hundred thousand, fully utilizing CPU and freeing memory.

Conclusion

Threads hit a scalability ceiling at the ten‑thousand‑connection level; coroutines raise that ceiling to the million‑level for I/O‑bound workloads. The choice between stackful and stackless designs depends on language philosophy and workload characteristics, and developers must be aware of the associated trade‑offs.