Understanding Application Service Avalanche and How to Prevent It

In distributed systems, service availability is never 100% due to network instability; when a service becomes unresponsive it can block callers and trigger a cascading avalanche effect.

Cache avalanche occurs when a cache server restarts or many cached items expire simultaneously, overwhelming backend databases and causing application‑server failures.

Scenarios that can cause an avalanche

Traffic spikes caused by abnormal traffic or user retries.

Cache refresh storms where a sudden influx of requests exceeds the target service’s capacity.

Program bugs such as infinite loops or memory leaks.

Hardware failures like server crashes, power outages, or fiber cuts.

Severe database bottlenecks, e.g., long‑running transactions or SQL timeouts.

Thread‑synchronization waits where a core service calls a non‑core service that hangs, eventually exhausting the thread pool.

Cache‑avalanche mitigation

Typical cache‑expiration scenarios:

Cache server failure.

Partial expiration during peak periods.

Hot‑key expiration.

Solutions include:

Stagger cache TTLs by using different expiration times for different keys.

Introduce mutex locks to control database access while rebuilding the cache.

Deploy highly available cache clusters (e.g., Redis clusters).

Overall avalanche mitigation strategies

Three main protection mechanisms for dependent services:

(1) Circuit‑breaker pattern

Inspired by electrical fuses, it stops calls to a slow or failing service after a configurable error threshold, returning immediately and freeing resources; normal operation resumes automatically when the service recovers.

Key monitoring metrics

CPU load and usage.

Memory consumption.

MySQL long‑running transactions.

SQL timeout occurrences.

Thread count.

(2) Isolation pattern

Isolates request types into separate “islands” so that a failure in one does not affect others; commonly implemented with dedicated thread pools or separate service instances for critical components.

(3) Rate‑limiting pattern

Pre‑emptively caps QPS for each request type; requests exceeding the threshold are rejected early, preventing overload but not solving downstream dependency issues.

Circuit‑breaker design details

Based on Hystrix, it consists of three modules: request‑decision algorithm, recovery mechanism, and alarm.

Decision algorithm uses a lock‑free circular queue with 10 one‑second buckets, tracking success, failure, timeout, and reject counts; trips when error rate >50% and >20 requests in the last 10 seconds.

Recovery attempts a trial request every 5 seconds; if latency <250 ms the circuit closes.

Alarms log tripped events and trigger alerts when thresholds are exceeded.

Isolation design details

Two common isolation methods:

Thread‑pool isolation: each dependent service gets its own thread pool, allowing burst handling by queuing excess requests.

Semaphore isolation: an atomic counter limits concurrent threads; excess requests are immediately rejected.

Timeout mechanism design

Two timeout types are considered:

Waiting timeout: when a task is queued, its enqueue time is recorded; if it exceeds a configured limit it is discarded.

Execution timeout: rely on the thread‑pool’s get method with a timeout.

Early detection of avalanche

Continuously monitor key metrics; when they approach or exceed predefined thresholds, raise alerts to intervene before a full‑blown avalanche occurs.

Overall, the article provides a comprehensive overview of avalanche scenarios in application services and presents practical technical solutions.