Multi-Level Cache Architecture: Caffeine + Redis to Defeat Penetration, Breakthrough & Avalanche

Microservice high-concurrency bottlenecks often lie in cache architecture. Relying solely on Redis cannot withstand flash-sale hotspot traffic due to network I/O overhead; when traffic spikes, Redis CPU saturates, P99 latency spikes, interfaces time out, and database pressure surges.

The standard production solution is a two-tier architecture: L1 local cache (Caffeine) + L2 distributed cache (Redis) . L1 resides in JVM memory with zero network overhead, serving only the current service instance. L2 is a Redis cluster shared globally across instances, carrying most hotspot data.

Complete Read Path (Cache-Aside Pattern)

Query L1 Caffeine first; hit returns immediately with no network request.

On L1 miss, query L2 Redis; on hit, write back to local cache and return data.

If Redis also misses, query the database, then write results to both cache layers before returning.

Traffic is filtered through two cache layers; mature systems achieve over 90% cache hit rate , reducing database requests by an order of magnitude.

Real-World Case: An e-commerce product detail service during a major promotion saw Redis CPU frequently maxed out. After introducing Caffeine as L1, 75% of hotspot requests were intercepted locally , Redis cluster CPU dropped 60% , P99 latency improved significantly, and no timeout jitter occurred during the promotion.

However, multi-level caching is not a silver bullet. It introduces new challenges: data inconsistency across instances, and the classic cache penetration, breakthrough, and avalanche problems persist — and can be amplified if designed poorly.

Production Write Flow (Standard Microservice Sequence)

Critical Pitfall: Never update cache before database. If the DB write fails after cache update, dirty data results — a cause of real business losses.

Inside a transaction, update the database first to guarantee DB accuracy.

Delete the corresponding Redis key (not update).

Clear the current service instance's local cache.

Broadcast invalidation messages via Redis Pub/Sub or MQ to all other service instances to clear their local caches.

TTL expiration serves as a final safety net; even if messages are lost, caches expire automatically, ensuring eventual consistency.

Supplemental: Delayed Double-Delete — For scenarios demanding higher consistency, after the DB write wait for the maximum interface latency, then delete the Redis cache a second time to resolve brief dirty reads from concurrent read/write races.

Defeating the Three Classic Cache Problems with Layered Defense

① Cache Penetration: Querying Non-Existent Data

Symptom: Massive illegal IDs or malicious parameters bypass both cache layers and hammer the database.

✅ Parameter pre-validation — reject illegal parameters before cache lookup.

✅ Cache null values — when DB returns empty, store a null marker in both Caffeine and Redis with a short TTL to block repeated requests.

✅ Bloom filter — load all valid business primary keys into a Bloom filter; requests not present are rejected immediately, suitable for massive data volumes.

② Cache Breakthrough: Hotspot Key Expiration Under High Concurrency

Symptom: A super-hot product or config key expires; massive concurrent requests bypass cache and hit the database.

Multi-Level Advantage: Hot data often resides in local cache; even if the Redis key expires, L1 continues to serve traffic, inherently reducing breakthrough risk.

✅ Hot keys never expire — manage invalidation at the business layer, not via TTL.

✅ Mutex lock — on cache miss, allow only one thread to query DB and repopulate cache; others wait.

✅ Circuit breaker & fallback — under extreme DB pressure, return fallback data directly.

③ Cache Avalanche: Mass Simultaneous Expiration or Redis Outage

Symptom: Large batches of keys expire simultaneously, or the entire Redis cluster fails, flooding the database.

✅ TTL with random offset — e.g., 25–35 minutes random range to stagger expirations.

✅ Multi-level cache as natural fallback — even if Redis completely fails, local cache absorbs most traffic, preventing total business paralysis.

✅ Redis high availability — master-replica, Sentinel, or Cluster mode to avoid single points of failure.

✅ Service circuit breaking, rate limiting, and degradation — on Redis anomalies, return fallback data to protect the database.

Biggest Pain Point: Multi-Instance Local Cache Inconsistency

In multi-node deployments, after a data update, instance A clears its local cache while B and C retain stale data. Requests routed to different instances read conflicting old/new data — the most common pitfall in multi-level cache rollout.

Three production-ready solutions, choose by need:

Solution 1: Redis Pub/Sub Invalidation Broadcast

Best for small-to-medium projects. After data update, publish a cache invalidation event; all nodes subscribe and clear the corresponding local cache entry. Pros: Simple, no extra middleware. Cons: Message loss causes dirty reads; relies on TTL fallback.

Solution 2: MQ (RocketMQ/Kafka) Broadcast

Best for medium-to-large production systems. Send cache change events via RocketMQ/Kafka; higher reliability, supports retries, messages persist if instances are temporarily down. Pros: More reliable than Pub/Sub, supports retry.

Solution 3: Canal Listening to MySQL Binlog for Async Invalidation

Best for high-traffic complex businesses. Monitor database binlog directly; any data change triggers automatic cache deletion without business code involvement. Pros: Zero business-code intrusion; even direct DB bypasses (e.g., manual SQL updates) auto-invalidate cache. Cons: Heavier architecture, higher maintenance cost.

Key Insight: Most internet businesses accept millisecond-to-second eventual consistency. Strong-consistency domains (balances, inventory) should avoid multi-level caching — prioritize DB transactions for correctness; don't over-engineer for performance at the cost of data integrity.

Production Pitfall Checklist (High-Frequency Incident Points)

❌ No capacity limit on local cache — Caffeine must set max size (recommend ≤10% of JVM heap), enable Window TinyLFU eviction; otherwise unbounded hotspot influx causes OOM.

❌ Storing large objects in local cache — large objects exhaust JVM memory quickly; only cache lightweight hotspot data.

❌ Full local cache eviction (allEntries=true) — instant mass invalidation floods the database, triggering avalanche. Prefer precise single-key deletion.

❌ No cache monitoring — must monitor both layers' hit rates, key eviction counts, invalidation events, Redis latency. Abnormal hit-rate drops signal cache strategy bugs requiring immediate intervention.

❌ Not differentiating data types — static configs, reference data, product basics suit multi-level cache; inventory, balances, transactional data require caution with local cache.

❌ Treating local cache as reliable storage — local cache is only a hotspot accelerator; the database remains the source of truth.

Case Study Retrospective: Product Detail Service Transformation

Business Context

E-commerce product detail API with high QPS, previously using only Redis. During promotions, bestseller items caused high Redis pressure and occasional timeouts.

Transformation Plan

Product basic info cached in both tiers; inventory and price (strong-consistency fields) excluded from local cache, kept only in Redis .

Update flow: DB update → delete Redis key → MQ broadcast invalidation → all nodes clear local cache.

TTL with random offset to prevent collective expiration; hotspot keys managed actively by business, no auto-expiry.

Added null-value caching to block malicious ID penetration.

Results

Average interface RT dropped significantly, Redis cluster CPU pressure reduced, database access volume decreased sharply, and no interface jitter occurred during peak promotion periods. The team accepted second-level data inconsistency, fully aligning with business requirements.

Architect Summary

Multi-level cache architecture is not merely stacking two caches; it is a complete traffic-layered defense system .

L1 local cache intercepts the hottest traffic, eliminating network overhead; L2 Redis handles global hotspot sharing.

Read/write order must be strict: update DB first, then delete cache , using broadcast to resolve multi-instance local cache inconsistency.

Penetration, breakthrough, and avalanche require layered defense ; multi-level cache reduces risk but does not auto-solve them.

Accept eventual consistency ; refuse multi-level cache abuse in strong-consistency scenarios — never sacrifice business correctness for performance.

Launch with monitoring metrics — observe cache hit rates, dynamically adjust capacity and TTL per business needs.

Many teams see only performance gains after rolling out multi-level cache, ignoring consistency risks until sporadic dirty data appears in production. Performance optimization must never detach from business context; making the right trade-offs between performance and consistency is the key to successful architecture implementation.