This guide walks SRE and senior operations engineers through a real-world service‑avalanche incident, detailing alert hierarchy design, fault‑location commands, emergency SOPs, capacity‑baseline building, and post‑mortem best practices to dramatically reduce MTTR in distributed micro‑service environments.
This comprehensive SRE guide walks through a real-world service avalanche incident, detailing alert triggering, root‑cause analysis, step‑by‑step recovery, capacity baseline creation, layered alert design, automated scripts, and post‑mortem best practices to help engineers prevent and resolve large‑scale outages.
This article examines a Redis 3.x cluster failure caused by a master‑slave switch, detailing how improper Jedis timeout and retry settings triggered a service avalanche, and provides step‑by‑step analysis of the incident, code paths, and recommended configuration adjustments to prevent recurrence.
During a Redis 3.x cluster master‑slave failover, the default Jedis connection timeout of two seconds combined with six automatic retries caused each request’s Redis calls to accumulate up to sixty seconds of latency, triggering Nginx timeouts and a service‑avalanche, which was resolved by lowering timeout and retry settings.
A service‑wide avalanche occurred when a Redis 3.x master‑slave failover coincided with Jedis’ default 2‑second connection timeout and six retry attempts, causing up to 60‑second latencies; adjusting connectionTimeout, soTimeout to 100 ms and reducing maxAttempts to two limited latency to about one second and prevented cascade failures.
This article uses the historic Battle of Red Cliffs as an analogy to explain service avalanche, its causes in micro‑service architectures, and how circuit‑breaker, rate‑limiting, and isolation techniques can prevent cascading failures in modern distributed systems.
This article explains how Tencent Cloud MongoDB improves the maxTimeMS server-side timeout feature to prevent request backlog and service avalanche, covering native MongoDB limitations, optimizations in mongos write command support, and default configuration implementation.
Using the historic Red Cliffs battle as a metaphor, this article explains how linked services can cause a cascading failure—service avalanche—in microservice architectures, and details prevention techniques such as rate limiting, isolation, and especially circuit breaker mechanisms with their principles and recovery algorithms.
This article uses the historic Battle of Red Cliffs as an analogy to explain service avalanche in micro‑service architectures, analyzes its causes, presents real‑world scenarios, and details circuit‑breaker concepts, algorithms, recovery strategies, and practical mitigation techniques.
