Why Simple Retry Triggers Avalanches and How Smart Strategies Save 10M‑QPS Systems

Retry Strategy: From Simple to Intelligent

In the early hours, a minor latency increase from 15 ms to 200 ms in an account service caused a chain reaction: the RPC framework’s default "retry on failure up to three times" turned a few timed‑out requests into three‑fold traffic, overwhelming the downstream service and triggering a full‑site avalanche.

Who Retry Helps

Retry is meant only for transient, self‑healing faults such as brief network glitches, dropped packets, or a short GC pause. These failures are occasional and short‑lived, so a retry after a brief wait usually succeeds.

Deterministic errors (e.g., illegal parameters) and overload failures are not suitable for retry; the former will always fail, and the latter will be worsened by additional traffic.

Why Naïve Retry Fails

Two fatal issues arise at large scale:

Traffic amplification: when a downstream service fails, all incoming requests retry simultaneously, turning 10 k QPS into 30 k QPS and crushing the service.

Synchronous retry shock: retries fire at the same moment, creating a wave of traffic that hits the downstream before it can recover. In multi‑layer chains, the amplification multiplies (e.g., three layers each retrying three times can produce 27 requests from one original).

This uncontrolled amplification turns a harmless jitter into a system‑wide storm.

Backoff and Jitter

Exponential backoff spaces retries: wait 100 ms after the first failure, 200 ms after the second, 400 ms after the third, etc. This gives the downstream service breathing room.

Adding jitter randomizes each wait time, preventing a thundering‑herd effect when many requests fail together.

Idempotency: The Non‑Negotiable Prerequisite

Before retrying, the operation must be idempotent—multiple executions must produce the same result. Read operations are naturally idempotent; write operations need a unique idempotency key so the server can detect duplicates and return the original result without re‑executing.

Different failure types affect safety: connection refusals are safe to retry, explicit business errors are not, and timeouts are uncertain and require idempotency checks.

Retry Budget

Instead of granting every request unlimited retries, a token‑bucket limits total retry traffic (e.g., to 10 % of normal traffic). When the bucket is empty, retries are rejected, preventing uncontrolled traffic spikes during downstream outages.

Coordinating with Circuit Breaker and Timeout

When a circuit breaker opens, indicating systemic downstream failure, retries must be disabled and requests should fail fast.

Retries must also respect the end‑to‑end deadline; if the remaining time is insufficient for another attempt, the retry is abandoned.

Only one layer in a call chain should perform retries to avoid multiplicative amplification.

Hedged Requests

Beyond ordinary retries, hedged (or backup) requests are sent pre‑emptively when a request exceeds a latency percentile (e.g., P95). The faster of the two responses wins, and the slower is cancelled. This combats tail latency without waiting for a failure.

Hedging requires idempotent operations and is also limited by the retry budget to avoid excess traffic.

Putting It All Together

The intelligent retry system combines error classification, idempotency enforcement, exponential backoff with jitter, a token‑bucket retry budget, circuit‑breaker coordination, deadline awareness, and optional hedged requests. Each layer addresses a specific risk, and together they form a robust defense that scales to tens of millions of QPS without self‑inflicted overload.

However, such a full‑stack solution is overkill for small, shallow systems with low QPS and read‑heavy workloads; a simple single‑retry with backoff may suffice.

Ultimately, as call‑chain depth and traffic grow, retry evolves from a naïve instinct to a disciplined, budgeted, and tightly integrated component of the overall fault‑tolerance architecture.