Understanding and Solving Common Cache Problems: Consistency, Concurrency, Penetration, and More

Cache Consistency Issue

When data freshness is critical, cache entries must stay consistent with the underlying database and with each other across cache replicas. The usual practice is to update or delete the cache entry immediately when the source data changes, relying on expiration policies to prevent stale data.

Cache Concurrency Issue

After a cache entry expires, many concurrent requests may simultaneously query the database, creating a “snowball” load and possible consistency problems when the key is being refreshed. A common mitigation is to use a distributed lock (e.g., Redis SETNX, Zookeeper, etc.) around the database read/update. The lock is acquired before accessing the DB; other requests wait briefly or read a stale value until the lock is released.

Cache Penetration Issue

Cache penetration occurs when a frequently requested key is missing in the cache and the corresponding data in the database is also empty (null). High‑traffic requests therefore bypass the cache and hit the DB, generating unnecessary load.

Mitigation methods:

Cache empty results (e.g., an empty list or a placeholder object) with a short TTL.

Implement a request filter that records keys known to have empty results and returns the cached placeholder without querying the DB.

Cache Jitter (Oscillation) Issue

Cache jitter refers to transient performance degradation caused by cache node failures. Using consistent hashing to distribute keys evenly across nodes reduces the impact of a single node outage.

Cache Avalanche

A cache avalanche happens when many keys expire simultaneously, causing a massive surge of database queries that can overwhelm the backend.

Mitigation strategies include:

Staggered (randomized) TTLs for different keys to avoid synchronized expiration.

Rate limiting, circuit breaking, and graceful degradation at the application layer.

Multi‑level caching (e.g., local in‑process cache plus distributed cache) to absorb request spikes.

Cache Bottomless‑Pit Phenomenon

This phenomenon was first reported by Facebook engineers around 2010 when their Memcached deployment grew to thousands of nodes. Adding more nodes did not improve performance because connection‑frequency overhead and latency dominated, leading to a “bottomless pit” where scaling failed.

Modern systems use sharding (hash‑based or range‑based) to meet high‑performance, high‑concurrency, high‑availability, and scalability requirements, but each operation may involve network communication with multiple nodes, increasing overhead.

Optimization Recommendations

Data Distribution : Choose hash‑based sharding for uniformly distributed workloads and range‑based sharding for ordered queries to reduce cross‑node traffic.

I/O Optimization : Employ connection pools, non‑blocking I/O (NIO), and keep‑alive sockets to lower per‑connection cost and improve concurrent throughput.

Data Access Pattern : Batch reads—fetch larger data sets in a single request rather than many small requests—to minimize network round‑trips.