Mastering Traffic Control: Keeping Systems Stable Under a Million Concurrent Requests

When a flash‑sale or promotion launches, traffic can surge like a flood and cause the system to crash; simply adding machines or scaling out is often too slow to respond.

First technique – rate limiting: Four common algorithms are compared. A fixed window releases a set number of requests per second but can burst at window boundaries. A sliding window smooths traffic by counting in finer time slices, as used by the Sentinel framework. The leaky‑bucket algorithm drains requests at a constant rate, suitable for smoothing but adds latency. The token‑bucket algorithm allows bursts while maintaining a steady token refill rate; Guava’s RateLimiter implements this, and in distributed environments tokens are often managed via Redis to reduce network overhead. The author recommends the token‑bucket approach for most internet scenarios.

Second technique – circuit breaker: It protects downstream services by monitoring failure rates. The circuit breaker has three states: Closed (normal operation, counting failures), Open (failure rate exceeds a threshold, all calls are rejected and a cooldown timer starts), and Half‑Open (after cooldown, a few trial calls are allowed to test recovery). This state machine enables fast failure and graceful recovery, preventing a single downstream outage from collapsing the entire call chain.

Third technique – degradation (fallback): When traffic exceeds capacity, non‑essential functions are disabled to preserve core services. The three pillars are “cache is king” (serve reads from browser cache, CDN, Nginx, local cache, or Redis), “asynchrony is champion” (write requests are queued to a message broker, sacrificing immediacy for higher throughput), and “distribution is the general” (horizontal scaling, micro‑service decomposition, and data sharding by user or region). The guiding principle is “protect the core, drop the secondary”.

Practical three‑step template: 1) Deploy rate‑limiting (prefer token bucket) at API gateways or RPC clients, with thresholds determined by load testing and managed via a configuration center. 2) Configure circuit breakers for each downstream service, setting failure‑rate thresholds and cooldown periods. 3) Define degradation plans: prioritize core flows, route reads through caches, funnel writes to MQ, and enable one‑click switches for optional features.

Remember that rate‑limit thresholds need iterative tuning through stress tests and must remain dynamically adjustable. By mastering these three “axes”—rate limiting, circuit breaking, and degradation—systems can handle millions of concurrent requests with stability.