Cache Design and Optimization Strategies in High-Concurrency Distributed Systems

In high‑concurrency distributed systems, caching is essential for accelerating read/write operations and shielding backend services from overwhelming request loads.

Cache Benefits and Costs

Benefits include accelerated memory‑based reads/writes (e.g., Redis can handle tens of thousands of QPS versus a few thousand for MySQL) and reduced backend load by offloading expensive computations. Costs involve data inconsistency due to cache‑storage latency windows, increased code maintenance for handling both cache and storage logic, and operational overhead for ensuring high availability (e.g., master‑slave replication, clustering).

Cache Update Strategies

Three primary update policies are presented:

LRU/LFU/FIFO eviction – choose one algorithm when cache space is insufficient; LRU removes least‑recently‑used items, LFU removes least‑frequently‑used, FIFO follows insertion order. These have low development cost but limited consistency.

Timeout eviction – set an explicit expiration time (e.g., Redis EXPIRE ) so stale data is refreshed from the source after a defined window. Consistency is moderate; implementation cost is low.

Active refresh – proactively update cache when the underlying data changes, offering the highest consistency but higher development and operational complexity, often requiring message queues and fallback timeout strategies.

Best‑practice recommendations: combine the first two strategies for low‑consistency workloads, and combine the second and third for high‑consistency scenarios.

Penetration Optimization

Cache penetration occurs when queries for non‑existent data miss both cache and storage, potentially overwhelming the backend. Mitigation techniques include caching empty objects (with short TTL) and employing Bloom filters to quickly reject invalid keys.

Hole (No‑Bottom‑Hole) Optimization

The “hole” problem arises from distributed hash‑based key placement causing many network round‑trips during batch operations (e.g., MGET ). Solutions range from serial GET loops, serial I/O grouping by node, parallel I/O with multithreading, to using Redis hash‑tags to force related keys onto the same node, reducing network overhead.

Avalanche Optimization

Cache avalanche happens when a large portion of cached data expires simultaneously, flooding the storage layer. Prevention includes ensuring high cache availability (e.g., Redis Sentinel), employing circuit‑breaker/limiters (e.g., Netflix Hystrix), and isolating resources per project.

Hot‑Key Rebuild Optimization

When a hot key expires, many threads may concurrently rebuild the cache, causing spikes in backend load. Strategies include using a mutex to allow only one rebuild thread (others wait or serve stale data), making certain keys never expire and updating them via scheduled jobs, and applying backend rate‑limiting to throttle rebuild attempts.

Conclusion

The author shares practical insights on cache design, selection of eviction policies, and advanced techniques to handle penetration, hole, avalanche, and hot‑key scenarios, emphasizing a balance between performance gains and operational complexity.