Preventing Avalanche Effect in Distributed Storage Systems: Replication Strategies, Flow Control, and Safety Mode

1. Background of Distributed Storage Systems

Replication is a common concept in distributed storage: data is stored in multiple copies according to a redundancy policy to ensure availability during local failures.

Two typical replication methods are used: (1) Pipeline – a→b→c, high throughput but suffers from slow‑node bottlenecks; (2) Distribution – client→a, client→b, client→c, lower throughput but avoids slow‑node issues. The article adopts a three‑replica scheme.

Automatic replica recovery works when a node fails, but large‑scale failures (e.g., high disk or switch failure rates) can cause many simultaneous recoveries, stressing the cluster.

2. Origin of the Avalanche Effect

When many nodes fail within a short period, the system may launch massive replica‑completion processes. Two factors make this dangerous: (a) overall free space is low (often ≤30% globally, ≤20% locally); (b) mixed‑deployment of applications on the same physical/virtual machines.

Cloud‑storage services often operate near capacity to reduce costs, so a burst of replica repairs can quickly fill remaining quota, leading to further node failures and a cascading avalanche.

3. Preventing the Avalanche

This section discusses internal logic improvements to avoid system‑wide collapse, illustrated with real‑world cases.

Case 1: Cross‑Rack Replica Selection and Resource Isolation

During a sudden loss of dozens of machines, engineers temporarily reduced the replica‑repair threshold from 3 to 2, fixed the network switch issue, and later restored normal parameters after the cluster recovered.

Improvement measures include adding hot‑fix support to the master, implementing a cross‑rack (or cross‑switch) replica‑placement algorithm, and partitioning machines and users by region to limit fault impact.

Case 2: Cluster Flow Control

General principle: no operation should consume excessive processing time, especially during traffic spikes or partial failures. Strategies involve user‑level flow control, token‑based node‑level flow control, and dedicated GC flow control.

Additional measures: flow‑control blacklists for abusive users, limiting concurrent replica repair/creation, and prioritizing operations based on resource consumption.

Case 3: Predictive Actions

Predict disk failures and proactively migrate data from at‑risk disks; add single‑disk fault tolerance. Predict load imbalance and perform pre‑emptive rebalancing, while balancing complexity against optimization benefits.

4. Safety Mode

When the number of failed nodes exceeds a configured threshold within a time window, the cluster enters safety mode, halting replica repair, reads, and writes until the situation is resolved.

Safety mode protects the system but requires careful tuning of thresholds, actions, and recovery procedures based on workload characteristics.

5. Reflection

The article only covers a limited set of scenarios; real distributed storage systems are far more complex. Designers must balance automation, flow control, latency, and resource overhead while considering user‑level isolation and regional partitioning.