How Simple Retry Can Crash Your System and Smarter Alternatives

Simple Retry Mechanism

The most obvious fault‑tolerance approach is "retry on failure", which is simple to implement but can cause a "snowball" effect because each retry doubles the request load on backend services.

If a service normally succeeds 99.9% of the time but drops to 95% due to a transient issue, a single retry can raise the effective success rate to about 99.75%. However, when the service truly fails, the extra traffic may overwhelm the system, especially as users repeatedly click a failing feature, amplifying the load.

Simple retries should be applied only in appropriate scenarios; otherwise, calculate the service success rate and disable retries when the rate is too low to avoid excessive traffic spikes.

Active‑Standby Service Auto‑Switching

Instead of retrying the same service, use two redundant services: if Service A fails, automatically request from Service B. This shifts the retry load to the standby service without doubling traffic on the primary.

Potential issues include resource waste (standby machines idle most of the time), increased latency (failed primary request followed by standby request doubles response time), and the risk that both primary and standby become unavailable under heavy load.

Dynamic Removal or Recovery of Unhealthy Machines

Backend services are deployed statelessly across many machines and routed through a common intelligent routing layer (L5). When a machine’s success rate falls below 50%, the router automatically removes it and periodically probes to reinstate it once healthy.

This self‑healing approach reduces manual intervention and improves overall system resilience.

Setting Reasonable Timeouts

Choosing an appropriate timeout for service calls is crucial. Overly long timeouts waste resources, as workers wait idle during network or service delays, reducing throughput. Conversely, overly short timeouts increase failure rates for legitimately slow operations.

Two mitigation strategies are suggested:

Fast‑slow separation: configure different timeout values for fast and slow services based on their typical latency.

Asynchronous I/O multiplexing: use coroutines or event‑driven models to avoid blocking threads during I/O waits, improving throughput without sacrificing latency.

Anti‑Reentrancy and Preventing Duplicate Deliveries

In scenarios like gift‑package delivery, short timeouts can cause "timeout‑success" cases where the user sees a failure but the backend later completes the operation, leading to repeated clicks and duplicate deliveries.

Solutions include:

Business‑level limits: restrict each user to a single package.

Order‑number mechanism: associate each delivery request with a unique order ID to ensure one‑time execution.

Asynchronous retry delivery: acknowledge success immediately to the user while processing delivery in the background, though this may affect user experience.

Special anti‑abuse measures: limit the number of read‑timeout occurrences per user to curb repeated attempts.

Overall, the article emphasizes that fault‑tolerance involves a combination of retry strategies, dynamic node management, proper timeout settings, asynchronous processing, and safeguards against duplicate actions.