Mastering High-Concurrency Caching: Penetration, Breakdown, Avalanche & Hot Keys

How Caching Works: Cache-Aside Pattern

The article starts by illustrating the standard read flow using the Cache-Aside pattern:

Read request → Check Redis?
├── Hit → Return directly
└── Miss → Query DB → Refill cache → Return

The core value of caching is that databases handle high concurrency poorly while Redis excels at it. However, caching shifts the problem upstream: any read that Redis fails to absorb hits the database directly, creating pressure. The four classic failure modes—penetration, breakdown, avalanche, and hot keys—are all scenarios where Redis "fails to catch" the traffic.

1. Cache Penetration

Definition: Querying a non-existent key; cache never hits, every request goes to DB.

Harm: Malicious attacks with non-existent IDs can overwhelm the DB.

Solutions:

Empty-value caching (simplest): When DB returns null, cache an empty value with a short TTL (e.g., 60 seconds). Example code:

Object v = redis.get(key);
if (v == null) {
    Object db = db.get(key);
    if (db == null) redis.set(key, "", 60); // cache empty value
    else redis.set(key, db, 3600);
    v = db;
}

Bloom filter: Pre-check if a key "might exist"; reject immediately if not. Note: has a false-positive rate (may say exists when it doesn't), but never false negatives.

Interface-layer validation: Block illegal/out-of-range parameters early.

2. Cache Breakdown (Hotspot)

Definition: A hot key expires exactly at a moment of high concurrency; many concurrent requests find the cache empty and all hit the DB simultaneously.

Difference from penetration: Penetration = key does not exist; Breakdown = key exists but just expired + sudden concurrent surge.

Solutions:

Mutex lock (most common): Only one thread rebuilds the cache; others wait. Example:

String v = redis.get(key);
if (v == null) {
    String lock = "lock:" + key;
    if (!redis.tryLock(lock, "1", 3)) { // lock acquire failed
        Thread.sleep(50); return retryGet(key); // wait and retry
    }
    try { v = db.get(key); redis.set(key, v, 3600); }
    finally { redis.unlock(lock); }
}

Logical expiration: Cache entry stores an expiration timestamp internally; no TTL set in Redis. On read, if expired but data exists → trigger async background rebuild, return stale data immediately. Suitable for read-heavy, short-staleness-tolerant scenarios.

3. Cache Avalanche

Definition: Large numbers of keys expire simultaneously (or Redis goes down), causing a flood of requests to hit the DB, potentially crashing it and triggering a cascading failure loop.

Harm: DB collapse can lead to a vicious cycle of "service avalanche" following "cache avalanche".

Solutions:

TTL jitter (staggered expiration): Add random offset to TTL, e.g., int ttl = 3600 + new Random().nextInt(600); High availability: Redis clustering/replication to avoid single point of failure; local cache (e.g., Caffeine) as a second-level fallback.

Rate limiting & circuit breaking: Protect DB with rate limiters and fallback degradation.

Persistence: Enable Redis persistence (RDB/AOF) to recover data after restart, reducing the "cold-start penetration" window.

Cache warming: Pre-load hot data before major promotions or after restarts to shrink the "all-miss" window.

Quick Comparison & Pre-Launch Checklist

Problem comparison:

Penetration — Essence: Key does not exist — Core countermeasure: Empty-value caching / Bloom filter

Breakdown — Essence: Hot key expires + concurrent surge — Core countermeasure: Mutex lock / Logical expiration

Avalanche — Essence: Mass keys expire together — Core countermeasure: TTL jitter + clustering + rate limiting

Pre-launch checklist:

[ ] TTL includes random jitter? (Bulk writes easily expire together)

[ ] Penetration protection in place? (Empty-value cache or Bloom filter, pick at least one)

[ ] Hot-key rebuild lock logic correct? (Use SET NX EX three-parameter form)

[ ] Redis persistence enabled? Cluster/replication configured?

[ ] DB-level rate limiting & circuit breaking configured?

4. Hot Keys & Multi-Level Caching (The Fourth Problem)

Beyond the three classic issues, a single key can receive extremely high traffic (e.g., flash-sale items, trending topics). Redis clusters shard by key hash, so a hot key saturates one shard while others idle—scaling out doesn't help because traffic still hits the same node.

Solutions:

Local cache fallback (multi-level caching, most effective): Add an in-process cache (e.g., Caffeine) in front of Redis. Read path becomes: Local → Redis → DB. A local hit avoids Redis entirely. Example configuration:

Cache<String, Object> local = Caffeine.newBuilder()
    .maximumSize(10_000)
    .expireAfterWrite(5, TimeUnit.SECONDS) // second-level TTL, tolerate 5s staleness
    .build();

Boundary: Nodes invalidate at different times, so short-term inconsistency is inherent . Suitable for price display, configs, homepage recommendations; avoid for strong-consistency fields like inventory balances or coupon quotas.

Hot key sharding: Split a hot key into multiple suffixes (e.g., sku:1001:0 … sku:1001:N), read randomly from one shard to distribute load across nodes.

Hot key detection: Must detect sudden hot keys first—use redis --hotkeys, client-side access stats, or gateway-layer TopK counting; then apply multi-level caching or sharding.

5. Cache-Database Consistency (The Mandatory Finale)

Using cache inevitably raises the question of when to update it. The recommended Cache-Aside pattern:

Write: Update DB first, then delete cache (recommended)
Read: On miss, query DB and refill cache

Why not delete cache first / write cache first? Leads to "stale data overwrites fresh data" or "concurrent reads see old values" inconsistencies.

Update DB then delete cache, combined with "delayed double delete" (delete again after ~500 ms) eliminates the vast majority of concurrent inconsistency windows.

Strong consistency requirement: If business truly needs strong consistency, don't use cache—read DB directly (or use distributed lock + message synchronization).

Interview / Production FAQs

Why is empty-value caching enough for penetration, why Bloom filter? → Empty-value caching is simple and effective for small key spaces; Bloom filter saves space when keys are massive or malicious.

How to distinguish breakdown from penetration? → Breakdown = key exists but expired; Penetration = key never existed.

Mutex lock vs logical expiration? → Strong consistency → mutex lock; read-heavy, tolerate staleness → logical expiration.

How to detect and handle hot keys? → Detect first (hotkeys / gateway TopK), then apply multi-level caching or sharding.

Cheat Sheet Summary

Root cause of all issues: caching front-loads read pressure; any traffic "not caught by Redis" slams the DB.

Three brothers mnemonic: Penetration → prevent "query nothing" (empty-value/Bloom); Breakdown → prevent "expiry instant" (mutex/logical expiration); Avalanche → prevent "mass expiry" (jitter/cluster/rate-limit).

Hot keys → multi-level caching + sharding; local cache must have short TTL + size limit; avoid for strong-consistency fields.

Consistency main line: update DB first, delete cache, optionally delayed double delete.

Before big promotions / launches: warm hot data + run the prevention checklist.

Takeaway: Caching is a safety net for "correct high concurrency", not for "lazy consistency". Guard the four gates—penetration, breakdown, avalanche, hot keys—and Redis becomes your system's first line of defense, not its first landmine.