17-year internet software developer specializing in AI applications, networking, architecture, and open source. Led the delivery of network services handling hundreds of millions of concurrent devices and tens of millions of QPS, and has three years of experience designing and building an agent platform. Follow to stay updated.
The article explains how a shared thread pool can turn a localized slowdown into a system‑wide outage, then walks through the bulkhead isolation pattern, bounded queues, trade‑offs between thread‑pool and semaphore isolation, sizing formulas, monitoring metrics, and dynamic tuning for high‑QPS services.
An online gateway crashed despite low CPU and memory because its one‑connection‑one‑thread BIO model exhausted threads, prompting a deep dive into thread‑pool limits, epoll, Reactor patterns, Proactor, and back‑pressure, showing how each evolution decouples connections from resource consumption to achieve million‑QPS scalability.
The article examines why ultra‑high‑throughput systems must move from costly strong consistency to eventual consistency, explaining consistency layering, the trade‑offs of coordination latency, availability, and tail latency, and how local message tables, idempotent delivery, compensation (Saga) and periodic reconciliation together ensure data eventually aligns without sacrificing performance.
Scaling a service from millions to tens of millions of queries per second fails not because of data size but due to per‑request fixed costs, and the article shows how batching aggregates these costs, dramatically boosts throughput, reduces latency, and introduces new challenges such as memory pressure and partial failures.
The article explains how moving non‑critical actions in a high‑traffic e‑commerce order flow to a task queue separates core work from optional work, solving delivery guarantees, idempotency, backlog management, and ordering challenges while dramatically reducing latency and fault contagion.
The article explains how, at the ten‑million‑QPS scale, the hidden cost of thread‑bound synchronous calls—memory, scheduling overhead, and stability risks—explodes, making asynchronous architectures essential, and outlines the trade‑offs, gradual migration paths, and scenarios where async should or should not be applied.
The article explains how a naïve ‘retry three times’ default can amplify a minor 200 ms latency spike into a full‑scale outage in a 10 million‑QPS system, and walks through a progressive design—from identifying retry‑eligible transient faults, adding exponential backoff with jitter, enforcing idempotency, applying token‑bucket retry budgets, integrating circuit breakers, to using hedged requests—showing how each layer prevents traffic amplification and ensures reliable high‑throughput services.
A 3‑second fixed timeout turned an 800 ms latency spike in a recommendation service into a full‑site outage, illustrating how static timeouts can exhaust thread pools; the article walks through evolving from global timeouts to per‑interface limits, deadline propagation, and coordinated timeout‑retry‑circuit‑breaker strategies for resilient 10 M‑QPS systems.
A flame‑graph reveals that in high‑throughput services only about 30% of CPU time is spent on business logic while the rest is consumed by serialization, prompting a step‑by‑step evolution from flexible JSON to compact binary formats, zero‑copy techniques, and selective compression to keep costs under control at tens of millions of requests per second.
A midnight alert revealed latency jumping from 20 ms to 200 ms despite healthy CPU and memory, leading to an investigation that uncovered TCP connection handling—handshake delays, TIME_WAIT buildup, and CPU overhead—as the hidden bottleneck, and explains why short connections must evolve into pooled, globally managed connections at tens of millions of QPS.
