From Zero to Control: Implementing Traffic Shaping for 10 Million QPS Systems

What broke the system

A marketing teammate launched a midnight batch that read 20 million users from the database and pushed 20 million coupon messages into a topic in a few minutes. The downstream coupon‑consumer, which queried the user, the coupon template, wrote records and called risk‑control, hit the shared database at full speed, exhausting the connection pool and CPU. The overload cascaded to the order‑placement service, causing massive time‑outs for paying customers.

The key insight was that the problem was not the total volume but the fact that all 20 million messages arrived within three minutes, turning a burst into a sustained backlog that the queue could only buffer, not shape.

Why "peak‑shaving" is insufficient

Many assume a message queue automatically smooths traffic, but it only provides buffering. It cannot control the injection rate at the producer nor the consumption rate at the consumer. Uncontrolled injection turns a burst into continuous pressure; uncontrolled consumption can cause "revenge" consumption that overwhelms downstream databases, caches, and third‑party services.

Three failure points without shaping

Downstream fragility amplification : Consumers forward the entire backlog to downstream services, causing the weakest link to fail first.

Resource contention chain reaction : Low‑priority jobs (e.g., coupon sending) can steal CPU, connection pools, and I/O from high‑priority jobs (e.g., order processing), starving them.

Second‑avalanche during recovery : After a downstream outage, consumers race to catch up, often at 5‑10× the normal rate, triggering a repeat crash.

All three stem from uncontrolled rate at producer, consumer, and recovery phases.

Two basic rate‑limiting paradigms

Leaky bucket : A bucket with a small hole releases tokens at a constant rate; excess input is dropped. It guarantees a smooth, constant output but disallows bursts, even when downstream is idle.

Token bucket : Tokens accumulate at a fixed rate; a message can pass only when a token is available. This allows limited bursts while keeping the average rate under control.

Choosing between them depends on whether the downstream can tolerate bursts.

Where to place the “valve”

Producer‑side limiting : Insert a limiter before messages enter the queue (e.g., throttle the coupon‑sending loop to 2 k msg/s). It prevents any backlog but requires every producer to be disciplined.

Broker‑side limiting : The queue service enforces a global quota, providing a centralized safeguard that does not rely on producer compliance. It demands sophisticated implementation in the broker.

Consumer‑side limiting : Apply a limiter when pulling or processing messages, protecting downstream services. It can be tuned per downstream dependency but cannot stop backlog from growing.

In practice, a layered defense—producer for known batch jobs, broker for system‑wide quotas, and consumer for fine‑grained downstream protection—is common in trillion‑QPS systems.

From static thresholds to adaptive shaping

Static limits (e.g., 1 000 msg/s) assume downstream capacity is constant, which is rarely true. Day‑night load variations, promotional spikes, and scaling events make a fixed number either wasteful or dangerous.

Adaptive shaping borrows TCP’s AIMD principle: increase the rate gradually when downstream latency is low, and sharply reduce it when latency, error rate, or throttling signals rise. This turns the “valve size” decision into a feedback‑driven control loop.

Adaptive shaping eliminates the need for a perfect static value but adds complexity: signal collection, algorithm design, oscillation avoidance, and safety cages (hard upper/lower bounds).

What to do with throttled messages

Hold in place : Let messages stay in the queue and be consumed later—suitable for non‑time‑critical tasks.

Discard : Drop messages that have lost value (e.g., stale market data).

Degrade via a side‑path : Route messages to a low‑priority pipeline or storage for later replay.

Back‑pressure to upstream : Propagate throttling upstream, eventually rejecting or rate‑limiting the original request.

The choice depends on the business value decay of the messages.

Putting it all together

Starting from a self‑inflicted outage, the article clarifies that queues only buffer, not shape, identifies three rate‑loss failure points, introduces leaky‑bucket and token‑bucket limiters, evaluates placement at producer, broker, and consumer, evolves from static to adaptive shaping, and finally discusses handling of throttled messages. The core principle is that traffic shaping matches the output rhythm to downstream capacity rather than merely limiting throughput.