Skip to content
Redis Cache Penetration, Breakdown & Avalanche: Three Lines of Defense

Redis Cache Penetration, Breakdown & Avalanche: Three Lines of Defense

This article dissects Redis cache penetration, breakdown, and avalanche as three forms of cache-miss storms, detailing layered defenses — Bloom filters for penetration, mutex locks and logical expiration for breakdown, TTL jittering plus multi-level caching and rate limiting for avalanche — with implementation code, capacity formulas, and production checklists.

Architecture Pony
Architecture Pony
Architecture Pony

The article opens by framing cache penetration, breakdown, and avalanche not as three distinct failures but as three manifestations of the same root cause: cache misses triggering database backflow. The fragility lies at the seam between cache and database, where hit-rate collapse funnels all traffic to the database in a single instant.

1. Three Failure Scenes

Penetration: Attackers scan 50,000 random UUIDs per second against an order-detail endpoint. None exist, so every request misses cache and hits the database. The database returns empty, and the application never caches the empty result, so each repeat UUID triggers another database query. Within 30 minutes the 200-connection pool is exhausted by "empty queries," blocking legitimate orders. No data is stolen; the connection pool is simply used as a target.

Breakdown: A flash-sale item caches its detail for 60 seconds at 60,000 QPS. The moment it expires, 60,000 concurrent requests all miss cache and hit the database. The single-table limit is 10,000 QPS; the database is pierced, slow queries pile up, the connection pool drains, and a dozen unrelated databases on the same instance are dragged down. Recovery takes 20 minutes. The hallmark of breakdown is "collapse in one instant," but that instant carries the full traffic volume the cache normally absorbs.

Avalanche: A nightly batch job refreshes 100,000 product-list keys with SETEX 3600 at 00:00. Every day at 01:00 all 100,000 keys expire simultaneously. At 01:00:01, all list-page requests backflow, driving database QPS from 2,000 to 45,000. Slow-query alerts fire for 6 minutes. Periodic, rhythmic slowdowns almost always trace to bulk key expiration — the easiest avalanche to fix.

2. Technical Analysis: Three Defense Lines

2.1 Penetration: Three-Layer Defense with Priority

Root cause: non-existent data bypasses cache. Defenses are layered before backflow, ordered by cost:

Input validation — reject malformed/out-of-range parameters (negative IDs, over-length strings). Near-zero cost; stops most low-level attacks.

Cache empty values — when database returns empty, write a placeholder with short TTL (1–5 minutes). Subsequent identical illegal requests hit cache. Risk: key explosion if attackers craft infinite unique keys; mitigate by limiting empty-key count or rate-limiting empty writes.

Bloom filter — probabilistic "key may exist?" check before cache. Non-existent keys are rejected without touching cache or database. Ideal when the full set of legal keys is known and finite (product IDs, order IDs).

Layers are additive, not mutually exclusive. Validation always applies. Decision rule: if legal keys are enumerable, use Bloom filter; otherwise rely on empty-value caching plus validation.

2.2 Bloom Filter: Trading Probability for Space

Structure: a bit array + k hash functions. Insert: hash key k times, set all k bits to 1. Query: same k hashes; if any bit is 0, key definitely does not exist; if all are 1, key "may exist." False positives only go one way: non-existent keys may be misjudged as existent; existent keys are never misjudged as non-existent.

This one-way property perfectly fits penetration defense: a false positive merely causes an extra cache/database lookup; real data is never lost. At 1% false-positive rate, 99% of penetration requests are blocked at the Bloom layer, reducing database pressure to 1% of attack volume. Space savings are dramatic: 100 million order IDs in a Set consume several GB; a Bloom filter needs ~114 MB. Trade-offs: no deletion (bits are shared), no enumeration, false-positive rate rises with fill ratio.

2.3 Breakdown: Two Routes — Mutex Lock vs. Logical Expiration

Core question: "At the moment of expiry, who backfills, and what do the others do?"

Mutex lock route: All requests race for a distributed lock ( SETNX). Winner backfills cache; losers wait briefly then re-read cache. Database receives exactly one request. Strongest correctness; cost is added latency for waiters.

Logical expiration route: Key never physically expires; expiry timestamp lives inside the value. On read, if not expired, return data; if expired, return stale data and asynchronously trigger rebuild. Zero wait for any request; latency stays flat. Cost: temporary stale reads.

Choice is fundamentally a consistency-vs-availability trade-off: mutex lock favors strong consistency (no stale reads) at the expense of latency; logical expiration favors availability (constant latency) at the expense of consistency. For flash-sale detail pages where seconds-old data is acceptable but latency spikes are not, logical expiration wins. For inventory where stale reads cause oversell, mutex lock is mandatory. Hidden difference: mutex lock requires the key to be rebuildable (key truly gone); logical expiration requires the key to live forever (consumes memory) — suitable only for a tiny set of ultra-hot keys.

Third approach often mentioned: hot-key replicas (split one key into key:0 … key:4, read randomly). Spreads expiry impact and read pressure across five keys. Mitigates severity but does not cure root cause: replicas can diverge (writes must sync multiple copies), and expiry remains simultaneous unless each replica jitters its own TTL. Best as a supplement for "read-extreme, write-rare" scenarios, not a primary breakdown defense.

2.4 Avalanche: Two Forms, Separate Countermeasures

Form 1: Mass key expiry. Core tactic is staggering — add random jitter to TTL (base ±10–20%). Batch jobs that bulk-load cache must especially jitter, because they naturally "write a batch in the same second." Deeper layer: pre-warming — load hot data before major promotions to avoid cold-start backflow.

Form 2: Cache instance crash. Staggering is useless — all keys vanish at once. Only three layers remain: multi-level cache (local cache absorbs one layer), rate limiting (protect database from death), fallback (return defaults or static pages). Mental model: when cache fully dies, database can handle only ~10% of normal traffic; rate limiting and fallback are not optional — they are the only survival path.

2.5 Unified Metric: Backflow Amplification Factor

Backflow Amplification Factor = Actual DB QPS ÷ Theoretical Minimum Backflow QPS at Cache Miss

. Normal ≈ 1. Penetration: factor = inverse of illegal-key repeat rate (can be infinite). Breakdown: factor = concurrency at expiry moment (flash sale can hit tens of thousands). Avalanche: factor = (simultaneously expiring keys) × (per-key access frequency). All defenses aim to push this factor back to ~1: Bloom/empty-value crush penetration factor; mutex lock crushes breakdown factor; TTL jitter flattens avalanche factor. Viewed through this single metric, three defenses are one mission: ensure every "must-backflow instant" triggers exactly one backflow.

2.6 Why the Database Always Falls First

The endpoint is database crash, but the bottleneck is the connection pool . Under cache hits, the database needs only a few hundred connections rotating. During backflow storm, every request holds a connection waiting for query return → queries slow → connections return slower → queue builds → application thread pool gets dragged → upstream services time out. This chain explains why Incident 2 dragged down a dozen unrelated databases: they shared the connection pool or the same application thread pool. Failure spreads laterally through resource pools, not through data.

This dictates rate-limiter placement: must limit before backflow, not in front of the database. By the time requests reach the database, the pool is already saturated; the limiter's own queries cannot get through. Correct mount point: the line of code that "detects cache miss and prepares to backflow" — count and reject there, preserving pool headroom. Remember the hierarchy: cache-layer rate limiting protects the database; application-layer thread-pool isolation protects the service itself; database max_connections is merely the last safety net.

3. Core Request Paths

3.1 Read Path with Bloom Filter

Bloom filter sits at the very front of the read path. Full journey:

① Input validation: reject malformed/out-of-range → 400 ② Bloom check BF.EXISTS order_bf key Returns 0 (definitely not exist) → return business empty, end Returns 1 (may exist, includes false positives) ③ Cache GET cache:key Hit (including empty placeholder) → return ④ Database query Data found → write cache (with TTL) → return No data → write empty placeholder SETEX cache:key 120 "" Return business empty

Step ④ "write cache even on empty" is the empty-value scheme, complementary to Bloom, not a replacement. Bloom blocks keys outside the known set; empty-value blocks keys inside the set that genuinely have no data yet (e.g., querying order detail the instant after placement). With both layers, penetration path narrows to "Bloom false positive + genuinely no data" — negligible volume. Critical detail: empty-value TTL must be far shorter than normal TTL — empty may be transient (data arriving soon); a long placeholder would block the late-arriving real data.

3.2 Mutex Lock Rebuild Timeline

At breakdown moment, 100 requests hit an expired key. Mutex lock splits them: 1 acquires lock, 99 wait. Full timeline:

T+0 100 requests detect cache miss T+0 Each attempts SET lock NX EX 10 T+0 1 succeeds (gets rebuild right) 99 fail, enter wait T+10ms Lock holder queries DB (50ms) T+60ms Write back cache, release lock T+60ms Waiters retry cache read T+60ms All hit, return (If lock holder crashes mid-way) T+10s Lock auto-expires T+10s New requests re-contend lock and rebuild

Three implementation details determine reliability: SET NX EX must be a single atomic command — separate SET and EXPIRE leaves a permanent lock if process crashes mid-way, causing all waiters to hang forever.

After acquiring lock, double-check cache — the previous holder may have just written it; skipping re-read causes duplicate backflow.

Lock TTL must exceed worst-case rebuild time — if rebuild takes 500ms but lock TTL is 200ms, lock expires first, second request enters and both rebuild, breaking mutual exclusion.

Waiters must retry with backoff and a hard limit; beyond limit, degrade or error — no infinite spinning.

3.3 Logical Expiration Read Path

Logical expiration moves "expiry" from Redis layer to application layer: key never expires physically (no TTL); value is an object carrying an expireAt field. Read path has zero blocking: not expired → return data; expired → return stale data and attempt to grab lock for async rebuild — only the lock winner submits rebuild task; losers don't even submit (someone else is building). No request on this path ever blocks ; latency curve is perfectly flat. Cost: window from expiry to rebuild completion returns stale data. Implied ops requirement: these immortal keys need active cleanup (delete keys for offline products), otherwise cache only grows.

4. Core Implementation: Four Key Code Segments

4.1 Bloom Filter Capacity Planning

Only two parameters: bit array size m and hash count k . Given expected capacity n and target false-positive rate p , optimal formulas:

<code>n = expected key count p = target false-positive rate m = -n * ln(p) / (ln2)² = optimal bit count k = (m/n) * ln2 = optimal hash count Example: n=100M, p=1% m ≈ 960M bits ≈ 114 MB k ≈ 7 False-positive 3%: m ≈ 79 MB, k ≈ 6</code>

Rule of thumb: ~9.6 bits per key per 1% false-positive; ~7.8 bits per key per 3%. Most common mistake: underestimating n — false-positive rate spikes sharply when fill ratio exceeds design. If designed for 50M but loaded with 100M, rate jumps from 1% to 20%+. Therefore n must be set to 6–12 month upper bound, preferably double the memory. This is why RedisBloom's BF.RESERVE requires explicit capacity: on overflow it auto-creates chained sub-filters rather than letting false-positive rate run away, but chained queries degrade performance.

4.2 Mutex Lock Rebuild Reference Implementation

<code>/* Pseudo-code: breakdown protection */ v = redis.get(key); if (v != null) return v; /* hit */ lock = "lock:" + key; /* NX+EX atomic in one command */ if (redis.set(lock, "1", "NX", "EX",10)) { try { v = redis.get(key); /* double-check */ if (v != null) return v; v = db.query(key); redis.setex(key, ttl, v); return v; } finally { redis.del(lock); /* must release */ } } /* Lock acquisition failed: wait+retry with backoff cap */ sleep(50 + rand(50)); v = redis.get(key); if (v != null) return v; return degrade(); /* cap exceeded → fallback */</code>

Four common bugs, each mapped to a production incident:

Separate SET and EXPIRE — process crash leaves immortal lock, all requests hang indefinitely.

Missing double-check — lock holder just wrote cache, you query DB again.

No backoff cap — 99 waiters spin in tight loop, CPU and Redis connections both exhausted.

Missing fallback branch — rebuild failure throws raw error to user.

All mutex-lock complexity lives in this snippet; every missing branch costs a life.

4.3 Logical Expiration Reference Implementation

<code>/* value structure: { data, expireAt }, no TTL */ o = redis.getObj(key); if (o == null) return null; /* truly absent */ if (o.expireAt &gt; now()) return o.data; /* not expired */ /* Expired: async rebuild, return stale first */ lock = "rebuild:" + key; if (redis.set(lock,"1","NX","EX",5)) { pool.submit(() -> { data = db.query(key); redis.setObj(key, wrap(data)); redis.del(lock); }); } return o.data; /* stale fallback, no block */</code>

Compared side-by-side with mutex version, difference is stark: read path has zero sleep and zero retry; every request grabs stale value and leaves. Rebuild runs in async thread pool; pool must be rate-limited (prevent mass simultaneous expiry from blowing up thread pool and database — logical expiration stops breakdown, not avalanche , they must be governed separately). Lock role changes: no longer "make others wait," but "ensure rebuild task submits only once," so lock TTL only needs to cover task-submission window — 5 seconds suffices. Stale fallback also demands data field always serializable — if stale value is half-written corrupt JSON, the whole path crashes; writes must guarantee atomic replacement.

4.4 Correct TTL Jittering

<code>/* Wrong: jitter ratio too large, expiry uncontrollable */ ttl = 3600 * rand(0.5, 1.5); /* Right: base + small jitter */ base = 3600; jitter = rand(0, 600); /* ±10 minutes */ ttl = base + jitter; /* Batch cache load: whole batch shares base */ /* Each key independent jitter, expiry points naturally spread */ for (key in batch) { redis.setex(key, base + rand(600), render(key)); }</code>

Jitter amplitude must be reverse-engineered from "expiry density." Goal: push "simultaneously expiring key count" below database's tolerable backflow rate. Example: 100K keys, DB tolerates 5,000 backflow QPS → 100K expiry points must be spread over at least 20 seconds → jitter window ≥ 20 seconds . Too small: ineffective (Incident 3's 6-minute alert persists). Too large: some keys expire prematurely, hit-rate suffers. Another practice: keys in same batch share "base + independent jitter" rather than each drawing a completely unrelated TTL — expiry points distribute evenly across the window while preserving batch semantics.

5. Advanced Topics: Seven Accounts You Must Calculate

5.1 Bloom Filter Deletion Problem

Bits are shared across keys; clearing one key's bits would "clear" other keys. Three engineering solutions:

Periodic full rebuild — when product offline rate is low (daily few per thousand), nightly rebuild a fresh Bloom from full valid dataset, atomic swap (rename or double-buffer).

Counting Bloom — each slot uses 4-bit counter instead of 1-bit; delete decrements counter. Space ×4.

Cuckoo filter — supports deletion with similar false-positive rate, but implementation is complex.

Decision rule: stable key lifecycle (products, order IDs) → native Bloom + periodic rebuild; frequent deletions → Cuckoo filter. Often overlooked option: if legal keys are auto-increment IDs, replace Bloom with range check — ID < max_id means possible existence, zero cost.

5.2 Bloom Filter Bootstrap & Incremental Sync

Bloom filter must have full set before traffic arrives — initialization is an engineering task: full DB scan to load all legal keys at launch (100M keys ≈ 20–30 minutes), gated by feature flag for gradual rollout; thereafter incremental sync via dual-write on business writes (write DB + BF.ADD). Three mandatory pitfalls:

Dual-write failure — DB write succeeds, Bloom write fails → that key forever "does not exist" (false negative). Must have compensation job to diff and rebuild.

Redis restart — Bloom is a regular data structure; persistence enabled → survives; pure-memory mode → must rebuild.

Multiple businesses sharing one Bloom — different key spaces must be isolated (prefix separation), otherwise they pollute each other's false-positive rate.

5.3 Mutex Lock TTL Dilemma

Lock TTL is a double bind: too short → rebuild incomplete, lock expires, second request enters duplicate backflow (mutex broken); too long → holder crashes, everyone waits in vain. Engineering fix: lease renewal — holder spawns background thread to periodically extend lock TTL (watchdog style); on completion, actively release; on crash, lock auto-expires within TTL. Second mitigation: lock-in double-check — even if mutex briefly fails, second entrant re-reads cache first; if first holder's backflow already wrote, it hits directly. Duplicate backflow confined to an extremely narrow window. Breakdown defense does not need a perfect distributed lock; "probabilistic mutex + double-check + fallback" triple layer is sufficient; chasing perfect lock introduces new complexity.

5.4 Logical Expiration vs. Mutex Lock Comparison Table

<code>Mutex Lock Logical Expiration Latency Rebuild jitter Constant Consistency Strong (no stale) Weak (short stale) Implementation Lock+wait+retry Immortal key+async Memory Normal Key never released Applicable Consistency-sensitive Latency-sensitive ultra-hot Risk Lock timeout dilemma Stale window + cleanup</code>

Two questions decide selection: ① Does reading stale data cause trouble (inventory, funds)? Yes → mutex lock. ② Can expiry-moment latency jitter be tolerated? Tolerable → normal TTL suffices; intolerable (flash-sale detail, banner) and stale harmless → logical expiration. Most businesses stop at ① (yes, trouble), so mutex lock is the default; logical expiration belongs only to the rare ultra-hot keys. Making logical expiration the default is over-engineering — immortal-key cleanup cost and stale-window consistency risk become permanent overhead.

5.5 Multi-Level Cache Division of Labor

Local cache (Caffeine/Guava) + Redis two-tier is standard avalanche armor, but roles must be clear:

Local cache holds "ultra-hot + second-level stale tolerable" data (configs, feature flags, homepage recommendation slots). Small capacity (MB), short TTL (seconds), inconsistency tolerated.

Redis holds "hot + needs cluster sharing" data. Large capacity, long TTL.

Key mechanism: TTL staircase — local 10s + Redis 10min. Local expiry falls back to Redis (not database); Redis crash → local still holds for a few seconds. Anti-pattern alert: stuffing all business data into local cache — multi-instance inconsistency, cache drift, invalidation broadcast complexity all surface. Local cache only earns its keep for "read-heavy, write-light, stale-tolerable" niche.

5.6 Hit-Rate Definition & Alert Thresholds

Hit-rate is not

INFO stats
keyspace_hits / misses

— that ratio counts "truly non-existent keys" as misses; penetration attacks drive it to floor, masking real business hit-rate. Correct method: application-level instrumentation: Business Hit-Rate = Hits ÷ (Hits + Legitimate Misses) , illegal requests counted separately. Alert heuristics: business hit-rate week-over-week drop 5% → warning; backflow QPS exceeds 50% of DB capacity → critical. Latter is avalanche precursor; former signals penetration or bulk expiry. Watch trends, not instants: nightly batch job's normal backflow spike must be baselined, not alerted every time (cry-wolf effect).

5.7 Production Default Combo

Three defenses merged into one default checklist:

Penetration — input validation everywhere; enumerable legal keys → RedisBloom (error_rate 1%, capacity 6-month ceiling), DB empty → empty placeholder TTL 2 min.

Breakdown — default mutex lock (atomic NX+EX, double-check, backoff cap, fallback); ultra-hot keys only → logical expiration.

Avalanche — all SETEX forced through utility method (built-in base+jitter), batch jobs jitter window ≥20s by density, core endpoints get local cache 10s, rate-limit at backflow layer (trigger at 50% DB capacity) + static fallback page.

Final discipline: all three defenses must have rehearsed "real failure" in test env; unrehearsed defense = no defense.

5.8 Warm-up & Drill: Two Checklists

Major-promotion warm-up: ① 24h ahead, load estimated TOP hot data into cache (TTL covers entire event + jitter); ② Rebuild Bloom from latest full set and verify count; ③ Recalculate rate-limit thresholds for peak traffic and push to backflow layer; ④ Confirm local-cache refresh jobs running; ⑤ Load-test "cache fully down" scenario, verify fallback chain activates. Value of checklist: turns "do three defenses exist" into an auditable action.

Quarterly drill: ① Actually kill cache instance, observe if rate-limit triggers on time, fallback page works; ② Simulate penetration with attack script, verify Bloom interception rate and empty-value caching; ③ Manufacture hot-key expiry, verify mutex lock "only one backflow"; ④ Post-recovery, sample-check cache-DB consistency. Both checklists share one principle: defenses must be validated when healthy; failure moment is only acceptance. Most teams can write the solution, configure thresholds, but never drill — real incident reveals fallback switch broken, fallback page 404.

6. Learning Path

Layer 1 (Reproduce): Write 100-thread stress script against expiring hot key, observe DB QPS spike (reproduce breakdown); write random-UUID "attack script," watch hit-rate collapse and backflow amplification (reproduce penetration).

Layer 2 (Defend): Install RedisBloom, calculate 100M/1% parameters, build filter, measure actual false-positive rate of BF.EXISTS; complete mutex lock's four bug fixes, concurrency-test verify "only one backflow."

Layer 3 (Govern): Reproduce "100K keys simultaneous expiry," run with and without jitter, compare DB QPS curves; simulate Redis full crash, verify rate-limit/fallback chain actually works.

Layer 4 (Quantify): Turn "backflow amplification factor" into a monitored metric, compute it for all three scenarios on production, identify the weakest defense line.

7. Interview Questions (with Answers)

Twenty Q&A pairs covering definitions, priorities, risks, Bloom mechanics, capacity planning, deletion, bootstrap, mutex lock details, lock TTL, logical expiration flow, selection criteria, avalanche forms, TTL jitter math, local/Redis split, hit-rate definition, monitoring differentiation, amplification factor, latency/consistency impact, rate-limit threshold setting, and production default combo. Each answer distills the article's core technical points.

8. Summary

Returning to the unified metric: backflow amplification factor . Penetration, breakdown, avalanche are three triggers of the same failure; defenses are three means to the same goal — ensure every "must-backflow instant" triggers exactly one backflow . Bloom uses probability to block non-existent requests at the door; mutex lock uses coordination to compress concurrent backflows into one; TTL jitter uses randomness to flatten batched expiries; and when cache truly dies, multi-level cache, rate limiting, and fallback — three measures "independent of Redis" — become the final levee. Cache solutions only solve "cache healthy" problems; the fallback plan for "cache unhealthy" is the watershed of architectural maturity.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Backend DevelopmentRedisDistributed LockRate LimitingBloom FilterCache AvalancheCache BreakdownCache PenetrationMulti-level CacheLogical ExpirationTTL Jitter
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.