Redis Cache Penetration, Breakdown & Avalanche: Java Solutions & Production Lessons
This article analyzes Redis cache penetration, breakdown, and avalanche — three high-concurrency failure modes — detailing root causes, solutions like Bloom filters, logical expiration, and multi-level caching, with production-ready Java code, real incident postmortems, and interview-focused technical comparisons.
Cache Penetration 🔍
Definition : Requested data exists neither in Redis nor in the database , causing every request to bypass the cache and hit the database directly, spiking DB load.
Root Causes:
Malicious attacks: hackers craft large numbers of non-existent IDs (e.g., ID=-1, ID=999999)
Business bugs: lax parameter validation allowing illegal values
Data out of sync: database record deleted but cache not cleared
Core Solutions:
Bloom Filter (preferred) : Pre-load all existing keys into a Bloom filter; requests check the filter first and are rejected immediately if absent.
Empty-value caching : For keys that return null from the database, cache a placeholder with a short TTL (e.g., 5 minutes).
API-layer validation : Validate input parameters (e.g., ID > 0, format checks).
Cache Breakdown 💥
Definition: A single hot key suddenly expires , and a burst of concurrent requests hits the database simultaneously, overwhelming it instantly.
Root Causes:
Flash-sale items, viral articles, hit videos — keys with extremely high access volume
Hot key has a TTL; expiration coincides with a traffic spike
Core Solutions:
Mutex lock (guarantees consistency) : Use Redis SETNX to acquire a lock; only one thread queries the database and rebuilds the cache, others wait and retry.
Logical expiration (high performance) : Hot key has no physical TTL; the value stores an expiration timestamp. A background async thread refreshes the cache.
Hot key never expires : For extreme hot keys, omit TTL entirely and update via scheduled background jobs.
Cache Avalanche 🌋
Definition: A large number of keys expire simultaneously , or the Redis cluster goes down entirely , causing all requests to flood the database and trigger a cascading collapse.
Root Causes:
Bulk data import with identical TTL for all keys
Redis master-slave failover or cluster failure making the service unavailable
Cache server resource exhaustion and crash
Core Solutions:
Add random offset to TTL : Add a random offset (e.g., 1–5 minutes) to each key's expiration to desynchronize expiry.
Redis high-availability cluster : Deploy master-replica + Sentinel or Redis Cluster to ensure service availability.
Circuit breaking & degradation : Use Hystrix/Sentinel to trip when DB load is excessive, temporarily disabling non-critical endpoints.
Multi-level caching : Local cache (Caffeine) + Redis + database, intercepting requests at each layer.
Three-Problem Comparison
Cache Penetration : Core characteristic — query non-existent data; typical trigger — malicious attack, illegal params; core solution — Bloom filter, empty-value cache
Cache Breakdown : Core characteristic — single hot key expires; typical trigger — flash sale, hit product, trending; core solution — mutex lock, logical expiration
Cache Avalanche : Core characteristic — mass keys expire together / Redis down; typical trigger — bulk import, cluster failure; core solution — random TTL, HA cluster, circuit breaker
Interview Bonus Points
Real project encountered cache avalanche : during a 618 promotion page, 120k product records were bulk-imported with a 1-hour TTL, causing DB CPU to hit 100% on the hour. Fixed by adding 0–300s random offset to each key's TTL.
For cache breakdown , prefer logical expiration because mutex locks cause massive thread blocking under high concurrency, while logical expiration needs only one background thread — better for flash-sale scenarios demanding ultra-low latency.
Bloom filters have a false-positive rate (~0.1%–1%), so they cannot fully replace empty-value caching; combining both works best.
Core Code Implementations
1. Cache Penetration: Bloom Filter (Guava)
import com.google.common.hash.BloomFilter;
import com.google.common.hash.Funnels;
import org.springframework.stereotype.Component;
import javax.annotation.PostConstruct;
import java.nio.charset.StandardCharsets;
import java.util.List;
@Component
public class ProductBloomFilter {
// Expected insertions 1M, false-positive rate 0.01%
private static final int EXPECTED_INSERTIONS = 1000000;
private static final double FPP = 0.0001;
private BloomFilter<String> bloomFilter;
@PostConstruct
public void initBloomFilter() {
// Initialize Bloom filter
bloomFilter = BloomFilter.create(
Funnels.stringFunnel(StandardCharsets.UTF_8),
EXPECTED_INSERTIONS,
FPP
);
// Pre-load all existing product IDs at startup
List<String> allProductIds = productMapper.selectAllProductIds();
for (String productId : allProductIds) {
bloomFilter.put(productId);
}
}
// Check if product ID might exist
public boolean mightContain(String productId) {
return bloomFilter.mightContain(productId);
}
}Technical highlights
Precise control of false-positive rate and memory footprint (1M entries + 0.01% FPP ≈ 1.7 MB)
Pre-load at startup avoids runtime construction overhead
2. Cache Breakdown: Mutex Lock (Redis SETNX)
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.util.concurrent.TimeUnit;
@Service
public class ProductService {
@Autowired
private StringRedisTemplate redisTemplate;
@Autowired
private ProductMapper productMapper;
private static final String CACHE_KEY_PREFIX = "product:";
private static final String LOCK_KEY_PREFIX = "lock:product:";
private static final int LOCK_EXPIRE_TIME = 10; // lock TTL 10s
private static final int CACHE_EXPIRE_TIME = 30; // cache TTL 30min
public Product getProductById(String productId) {
String cacheKey = CACHE_KEY_PREFIX + productId;
// 1. Check cache first
String productJson = redisTemplate.opsForValue().get(cacheKey);
if (productJson != null) {
return JSON.parseObject(productJson, Product.class);
}
// 2. Cache miss, acquire mutex
String lockKey = LOCK_KEY_PREFIX + productId;
boolean lockAcquired = redisTemplate.opsForValue()
.setIfAbsent(lockKey, "1", LOCK_EXPIRE_TIME, TimeUnit.SECONDS);
if (lockAcquired) {
try {
// 3. Double-check cache after acquiring lock
productJson = redisTemplate.opsForValue().get(cacheKey);
if (productJson != null) {
return JSON.parseObject(productJson, Product.class);
}
// 4. Query database
Product product = productMapper.selectById(productId);
if (product == null) {
// Empty-value cache to prevent penetration
redisTemplate.opsForValue()
.set(cacheKey, "", 5, TimeUnit.MINUTES);
return null;
}
// 5. Update cache
redisTemplate.opsForValue()
.set(cacheKey, JSON.toJSONString(product),
CACHE_EXPIRE_TIME, TimeUnit.MINUTES);
return product;
} finally {
// 6. Release lock in finally
redisTemplate.delete(lockKey);
}
} else {
// 7. Failed to acquire lock, wait 50ms and retry
try {
Thread.sleep(50);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return getProductById(productId); // recursive retry
}
}
}Technical highlights
Double-checked locking avoids duplicate DB queries
Lock TTL prevents deadlock finally block guarantees lock release
Empty-value cache combined with mutex solves both penetration and breakdown
3. Cache Breakdown: Logical Expiration (High-Performance)
import com.alibaba.fastjson.JSON;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
@Service
public class HotProductService {
@Autowired
private StringRedisTemplate redisTemplate;
@Autowired
private ProductMapper productMapper;
private static final String CACHE_KEY_PREFIX = "hot:product:";
private static final int LOGIC_EXPIRE_TIME = 30 * 60; // logical TTL 30min
// Thread pool for async cache refresh
private static final ExecutorService CACHE_REFRESH_EXECUTOR =
Executors.newFixedThreadPool(10);
// Cache data object with logical expiration timestamp
private static class CacheData<T> {
private T data;
private Long expireTime; // logical expiration timestamp
public CacheData(T data, Long expireTime) {
this.data = data;
this.expireTime = expireTime;
}
// Check if expired
public boolean isExpired() {
return System.currentTimeMillis() > expireTime;
}
}
public Product getHotProductById(String productId) {
String cacheKey = CACHE_KEY_PREFIX + productId;
// 1. Query cache
String cacheJson = redisTemplate.opsForValue().get(cacheKey);
if (cacheJson == null) {
return null; // hot key pre-warmed, should not reach here
}
CacheData<Product> cacheData = JSON.parseObject(cacheJson,
new TypeReference<CacheData<Product>>() {});
Product product = cacheData.getData();
// 2. Check logical expiration
if (!cacheData.isExpired()) {
return product; // not expired, return directly
}
// 3. Expired: async refresh
String lockKey = "lock:hot:product:" + productId;
boolean lockAcquired = redisTemplate.opsForValue()
.setIfAbsent(lockKey, "1", 10, TimeUnit.SECONDS);
if (lockAcquired) {
// Submit async task to update cache
CACHE_REFRESH_EXECUTOR.submit(() -> {
try {
// Query database
Product newProduct = productMapper.selectById(productId);
// Update cache with new logical expiration
CacheData<Product> newCacheData =
new CacheData<>(
newProduct,
System.currentTimeMillis() + LOGIC_EXPIRE_TIME * 1000L
);
redisTemplate.opsForValue().set(cacheKey,
JSON.toJSONString(newCacheData));
} finally {
redisTemplate.delete(lockKey);
}
});
}
// 4. Return stale data regardless of lock acquisition
return product;
}
}Technical highlights
Non-blocking: all requests return immediately, no thread waits
Async update: dedicated thread pool handles cache refresh, no impact on main thread
Hot key pre-warming: load hot data into cache at startup to avoid cold-start issues
4. Cache Avalanche: Random TTL Offset
// Add 0-300s random offset to each key's TTL
int baseExpireTime = 30 * 60; // base TTL 30min
int randomOffset = new Random().nextInt(300); // random offset 0-5min
int finalExpireTime = baseExpireTime + randomOffset;
redisTemplate.opsForValue().set(
cacheKey,
JSON.toJSONString(product),
finalExpireTime,
TimeUnit.SECONDS
);Technical Challenges & Solutions Summary
Bloom filter false positives : Hash collisions cause non-existent keys to be misjudged as existing. Solution: 1. Lower FPP (more hash functions, larger bit array) 2. Combine with empty-value cache as fallback. Best practice: Set FPP 0.01%–0.1%, rebuild Bloom filter periodically.
Mutex lock deadlock : Service crash leaves lock unreleased. Solution: 1. Set lock TTL 2. Use Redis Lua script for atomic lock release. Best practice: Lock TTL = 2–3× DB query time.
Mutex lock thundering herd : Many threads wait for lock release, then simultaneously hit DB. Solution: 1. Increase retry interval 2. Use striped locks 3. Prefer logical expiration. Best practice: Retry interval 50–100ms to avoid frequent retries.
Logical expiration stale reads : Async update window serves old data. Solution: 1. Shorten logical TTL 2. Use mutex lock for strong consistency on critical paths. Best practice: Non-core (product detail, article) → logical expiration; core (order) → mutex lock.
Redis cluster outage avalanche : Cache layer completely down, all traffic hits DB. Solution: 1. Redis Cluster + Sentinel 2. Local cache fallback 3. Circuit breaker & degradation. Best practice: Local cache with Caffeine, set max size and TTL.
Bulk import avalanche : Mass keys expire simultaneously. Solution: 1. Random TTL offset 2. Batch import 3. Import during low-traffic windows. Best practice: Random offset = 10%–20% of base TTL.
Cache warm-up cold start : Empty cache at startup floods DB. Solution: 1. Pre-load hot data at startup 2. Gradual traffic ramp-up 3. Degradation mechanism. Best practice: Pre-identify hot keys, async load into cache at startup.
Real Interview Simulation
Interviewer 👨💻:
“Good basics. Now a scenario design: explain Redis cache penetration, breakdown, avalanche , clearly differentiate them and give solutions. Go ahead.”
Candidate 🧑💻:
“All three are essentially cache failing to shield traffic, hammering the database , but triggers differ and so do solutions.
Quick distinction:
🔍 Penetration : Querying data that does not exist in DB at all , cache naturally empty, requests ‘penetrate’ through.
💥 Breakdown : A hot key just expired , massive concurrent requests race to rebuild cache, ‘breaking through’ the DB.
❄️ Avalanche : Mass keys expire in same window , or Redis dies outright, requests avalanche onto DB.
May I walk through each with diagrams?”
Interviewer 👨💻:
“Sure, draw and explain.”
Candidate 🧑💻:
“First, 🔍 Cache Penetration .
Example: attacker repeatedly requests id = -1, which doesn’t exist in DB nor cache, every request hits DB.
Two solutions:
Bloom filter : Pre-load all valid IDs. Incoming requests hit filter first; non-existent ones blocked at the gate 🚪.
Cache empty object : Even if DB returns null, store a short-lived placeholder (e.g., "") with 30s TTL. Next identical request caught at cache layer.
In production I use Bloom filter upfront + empty-value fallback , double insurance.”
Interviewer 👨💻:
“Clear logic. How about breakdown?”
Candidate 🧑💻:
“💥 Cache Breakdown .
Classic case: flash-sale item, key hot_item:123. When this hot key expires during peak traffic, all requests rush to DB to rebuild cache.
Core tactic: prevent concurrent rebuild.
Mutex (Mutex) : Use Redis SETNX for a lightweight lock; only one thread queries DB, others sleep tens of ms and retry cache.
Logical expiration : Hot key has no TTL; value embeds logical expiry timestamp. On logical expiry, return stale value immediately, spawn background thread to refresh. This is the high-performance king 🤖.
Pseudo-code sketch:
public String getHotData(String key) {
String value = redis.get(key);
if (value == null) {
if (redis.setnx(lockKey, "1", 10, TimeUnit.SECONDS)) {
try {
value = db.query();
redis.set(key, value, 3600);
} finally {
redis.del(lockKey);
}
} else {
Thread.sleep(50);
return getHotData(key);
}
}
return value;
}Interviewer 👨💻:
“If using logical expiration, how do you store the value?”
Candidate 🧑💻:
“Wrap in an object, e.g., HotDataWrapper, with two fields: data and expireAt (logical expiry timestamp). On read, check expireAt < now; if expired, return old data and spin a thread to acquire lock and rebuild.”
Interviewer 👨💻:
“Good. Now avalanche.”
Candidate 🧑💻:
“❄️ Cache Avalanche , highest destructive power.
Two overlapping scenarios: mass keys expire together, or Redis crashes.
Multi-pronged defense:
Randomize TTL with salt : Add 1–5 min random offset to base TTL, spreading expiry. Simplest and most effective 😊.
HA architecture : Redis master-replica + Sentinel/Cluster, eliminate cache single point of failure.
Circuit breaker & degradation : Sentinel/Hystrix trips when DB load spikes, returns degraded response (local cache, default, ‘busy’), combined with rate limiting.
Multi-level cache : Local Caffeine + Redis two-tier, Redis down still has local fallback.”
Interviewer 👨💻:
“Summarize with a comparison?”
Candidate 🧑💻:
“Table summary:
🔍 Penetration: Trigger — query non-existent data; Core idea — Bloom filter / empty object; One-liner — Prevent non-existent
💥 Breakdown: Trigger — hot key expires; Core idea — mutex / logical expiration; One-liner — Prevent rebuild race
❄️ Avalanche: Trigger — mass keys expire / crash; Core idea — randomize TTL / HA / degradation; One-liner — Prevent simultaneous down
Interviewer 👨💻:
“Nice. Follow-up: in a clustered deployment, if SETNX lock holder times out or node crashes, deadlock risk? How to solve?”
Candidate 🧑💻:
“Real deadlock risk. I add lock TTL and use Redisson’s watchdog auto-renewal — as long as thread lives, lock won’t expire; if node truly dies, lock auto-releases on TTL expiry. No stuck deadlocks.”
Interviewer 👨💻:
“Solid. Can you paste core code showing technical highlights, then summarize technical challenges & solutions for this whole scenario?”
Candidate 🧑💻:
“Sure, I’ll present by scenario with code, challenges, and insights. 😄”
🔍 Cache Penetration — Core Code
Technical highlights : Bloom filter + empty-value cache dual insurance, balancing performance and penetration defense.
// 1. Bloom filter init (Redisson, at startup)
RBloomFilter<String> bloomFilter = redissonClient.getBloomFilter(
"userIdFilter");
// Expected 1M inserts, FPP 0.01%
bloomFilter.tryInit(1_000_000L, 0.0001);
// Full ID warm-up, load from DB in batches
List<Long> allUserIds = userMapper.getAllIds();
allUserIds.forEach(id -> bloomFilter.add("user:" + id));
// 2. Query logic
public User getUserById(Long id) {
String key = "user:" + id;
// Bloom filter intercept
if (!bloomFilter.contains(key)) {
return null; // direct return, no DB hit
}
// Check cache
String cacheValue = redisTemplate.opsForValue().get(key);
if (cacheValue != null) {
// Check if empty placeholder
if ("NULL_PLACEHOLDER".equals(cacheValue)) {
return null;
}
return JSON.parseObject(cacheValue, User.class);
}
// Query DB
User user = userMapper.selectById(id);
if (user == null) {
// Cache empty object, 30s TTL to save space
redisTemplate.opsForValue().set(key, "NULL_PLACEHOLDER", 30, TimeUnit.SECONDS);
} else {
redisTemplate.opsForValue().set(key, JSON.toJSONString(user), 1, TimeUnit.HOURS);
}
return user;
}💥 Cache Breakdown — Core Code
Technical highlights : Redisson distributed lock + watchdog auto-renewal , avoids deadlock, ensures high concurrency safety.
public Product getHotProduct(Long id) {
String key = "hot_product:" + id;
String cacheData = redisTemplate.opsForValue().get(key);
if (cacheData != null) {
return JSON.parseObject(cacheData, Product.class);
}
// Distributed lock key
String lockKey = "lock:product:" + id;
RLock lock = redissonClient.getLock(lockKey);
try {
// Try lock, wait max 10s, lease 30s (watchdog renews, 30s fallback)
if (lock.tryLock(10, 30, TimeUnit.SECONDS)) {
// Double-check
cacheData = redisTemplate.opsForValue().get(key);
if (cacheData != null) {
return JSON.parseObject(cacheData, Product.class);
}
// Query DB
Product product = productMapper.selectById(id);
if (product != null) {
// Set cache with random TTL (300-420s) to prevent avalanche
redisTemplate.opsForValue().set(key, JSON.toJSONString(product),
300 + ThreadLocalRandom.current().nextInt(120), TimeUnit.SECONDS);
} else {
// Empty-value cache to prevent penetration
redisTemplate.opsForValue().set(key, "NULL_PLACEHOLDER", 30, TimeUnit.SECONDS);
}
return product;
} else {
// Lock acquire failed, sleep and retry
Thread.sleep(50);
return getHotProduct(id); // recursive retry
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return null;
} finally {
if (lock.isHeldByCurrentThread()) {
lock.unlock();
}
}
}❄️ Cache Avalanche — Core Code
Technical highlights : Random TTL salt + multi-level cache fallback (Caffeine).
// Local cache config (Caffeine)
Cache<Long, Object> localCache = Caffeine.newBuilder()
.maximumSize(1000)
.expireAfterWrite(60, TimeUnit.SECONDS)
.build();
public Object getDataWithMultiLevel(Long id) {
String key = "data:" + id;
// 1. L1: Local cache
Object localVal = localCache.getIfPresent(id);
if (localVal != null) {
return localVal;
}
// 2. L2: Redis
String cacheVal = redisTemplate.opsForValue().get(key);
if (cacheVal != null) {
localCache.put(id, JSON.parse(cacheVal)); // backfill local
return cacheVal;
}
// 3. Query DB with lock (prevent breakdown)
Object dbVal = queryFromDbWithLock(id);
if (dbVal != null) {
// Set Redis TTL: base 1h + random 0-600s
long expireSec = 3600 + ThreadLocalRandom.current().nextInt(600);
redisTemplate.opsForValue().set(key, JSON.toJSONString(dbVal), expireSec, TimeUnit.SECONDS);
localCache.put(id, dbVal);
}
return dbVal;
}🧩 Scenario Technical Challenges & Solutions
🔍 Penetration : Challenges — 1. Bloom filter cannot delete , requires rebuild on data changes. 2. False positives let some invalid requests through. 3. Empty-value cache consumes memory, may be abused with crafted keys. Solutions — 1. Use scalable Bloom filter (RedisBloom module) or scheduled rebuild. 2. Set FPP ≤ 0.01%, rely on empty-value cache as safety net. 3. Empty-value cache with ultra-short TTL, plus request rate limiting. Experience tip: Actual FPP must be stress-tested, not just theoretical.
💥 Breakdown : Challenges — 1. Lock granularity vs performance : more hot keys = heavier lock contention. 2. Logical expiration has eventual consistency — async update may return stale data. 3. Distributed lock deadlock risk . Solutions — 1. Single-node: ConcurrentHashMap + lightweight lock; cluster: Redisson fine-grained locks. 2. Ensure stale value remains usable, pre-load or background refresh. 3. Redisson watchdog auto-renewal + finally unlock. Experience tip: Logical expiration suits read-heavy, write-light workloads tolerating brief inconsistency.
❄️ Avalanche : Challenges — 1. Random TTL algorithm can still produce local hot-spot clusters. 2. Multi-level cache sync issues — local vs Redis may diverge. 3. Post-circuit-breaker recovery causes ‘pulse’ pressure spikes. Solutions — 1. Combine randomness with business-aware pre-warming of high-frequency keys. 2. Short local TTL, message queue for invalidation. 3. Half-open state gradual ramp-up via token bucket or warm-up algorithm. Experience tip: Production must have multi-level cache; core interfaces need stress-tested QPS limits and thresholds. 👍
Interviewer 👨💻:
“Excellent, code details spot-on, challenges broken down clearly. Your habit of linking code to solutions is a strong interview signal. This cache trio assessment earns you a Strong Hire ✅.”
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
