Skip to content
Redis Cache Penetration, Breakdown & Avalanche: Java Solutions & Production Lessons

Redis Cache Penetration, Breakdown & Avalanche: Java Solutions & Production Lessons

This article analyzes Redis cache penetration, breakdown, and avalanche — three high-concurrency failure modes — detailing root causes, solutions like Bloom filters, logical expiration, and multi-level caching, with production-ready Java code, real incident postmortems, and interview-focused technical comparisons.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

Cache Penetration 🔍

Definition : Requested data exists neither in Redis nor in the database , causing every request to bypass the cache and hit the database directly, spiking DB load.

Root Causes:

Malicious attacks: hackers craft large numbers of non-existent IDs (e.g., ID=-1, ID=999999)

Business bugs: lax parameter validation allowing illegal values

Data out of sync: database record deleted but cache not cleared

Core Solutions:

Bloom Filter (preferred) : Pre-load all existing keys into a Bloom filter; requests check the filter first and are rejected immediately if absent.

Empty-value caching : For keys that return null from the database, cache a placeholder with a short TTL (e.g., 5 minutes).

API-layer validation : Validate input parameters (e.g., ID > 0, format checks).

Cache Breakdown 💥

Definition: A single hot key suddenly expires , and a burst of concurrent requests hits the database simultaneously, overwhelming it instantly.

Root Causes:

Flash-sale items, viral articles, hit videos — keys with extremely high access volume

Hot key has a TTL; expiration coincides with a traffic spike

Core Solutions:

Mutex lock (guarantees consistency) : Use Redis SETNX to acquire a lock; only one thread queries the database and rebuilds the cache, others wait and retry.

Logical expiration (high performance) : Hot key has no physical TTL; the value stores an expiration timestamp. A background async thread refreshes the cache.

Hot key never expires : For extreme hot keys, omit TTL entirely and update via scheduled background jobs.

Cache Avalanche 🌋

Definition: A large number of keys expire simultaneously , or the Redis cluster goes down entirely , causing all requests to flood the database and trigger a cascading collapse.

Root Causes:

Bulk data import with identical TTL for all keys

Redis master-slave failover or cluster failure making the service unavailable

Cache server resource exhaustion and crash

Core Solutions:

Add random offset to TTL : Add a random offset (e.g., 1–5 minutes) to each key's expiration to desynchronize expiry.

Redis high-availability cluster : Deploy master-replica + Sentinel or Redis Cluster to ensure service availability.

Circuit breaking & degradation : Use Hystrix/Sentinel to trip when DB load is excessive, temporarily disabling non-critical endpoints.

Multi-level caching : Local cache (Caffeine) + Redis + database, intercepting requests at each layer.

Three-Problem Comparison

Cache Penetration : Core characteristic — query non-existent data; typical trigger — malicious attack, illegal params; core solution — Bloom filter, empty-value cache

Cache Breakdown : Core characteristic — single hot key expires; typical trigger — flash sale, hit product, trending; core solution — mutex lock, logical expiration

Cache Avalanche : Core characteristic — mass keys expire together / Redis down; typical trigger — bulk import, cluster failure; core solution — random TTL, HA cluster, circuit breaker

Cache flow diagram
Cache flow diagram

Interview Bonus Points

Real project encountered cache avalanche : during a 618 promotion page, 120k product records were bulk-imported with a 1-hour TTL, causing DB CPU to hit 100% on the hour. Fixed by adding 0–300s random offset to each key's TTL.

For cache breakdown , prefer logical expiration because mutex locks cause massive thread blocking under high concurrency, while logical expiration needs only one background thread — better for flash-sale scenarios demanding ultra-low latency.

Bloom filters have a false-positive rate (~0.1%–1%), so they cannot fully replace empty-value caching; combining both works best.

Core Code Implementations

1. Cache Penetration: Bloom Filter (Guava)

import com.google.common.hash.BloomFilter;
import com.google.common.hash.Funnels;
import org.springframework.stereotype.Component;
import javax.annotation.PostConstruct;
import java.nio.charset.StandardCharsets;
import java.util.List;

@Component
public class ProductBloomFilter {
    // Expected insertions 1M, false-positive rate 0.01%
    private static final int EXPECTED_INSERTIONS = 1000000;
    private static final double FPP = 0.0001;

    private BloomFilter<String> bloomFilter;

    @PostConstruct
    public void initBloomFilter() {
        // Initialize Bloom filter
        bloomFilter = BloomFilter.create(
            Funnels.stringFunnel(StandardCharsets.UTF_8),
            EXPECTED_INSERTIONS,
            FPP
        );

        // Pre-load all existing product IDs at startup
        List<String> allProductIds = productMapper.selectAllProductIds();
        for (String productId : allProductIds) {
            bloomFilter.put(productId);
        }
    }

    // Check if product ID might exist
    public boolean mightContain(String productId) {
        return bloomFilter.mightContain(productId);
    }
}

Technical highlights

Precise control of false-positive rate and memory footprint (1M entries + 0.01% FPP ≈ 1.7 MB)

Pre-load at startup avoids runtime construction overhead

2. Cache Breakdown: Mutex Lock (Redis SETNX)

import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.util.concurrent.TimeUnit;

@Service
public class ProductService {
    @Autowired
    private StringRedisTemplate redisTemplate;
    @Autowired
    private ProductMapper productMapper;

    private static final String CACHE_KEY_PREFIX = "product:";
    private static final String LOCK_KEY_PREFIX = "lock:product:";
    private static final int LOCK_EXPIRE_TIME = 10; // lock TTL 10s
    private static final int CACHE_EXPIRE_TIME = 30; // cache TTL 30min

    public Product getProductById(String productId) {
        String cacheKey = CACHE_KEY_PREFIX + productId;

        // 1. Check cache first
        String productJson = redisTemplate.opsForValue().get(cacheKey);
        if (productJson != null) {
            return JSON.parseObject(productJson, Product.class);
        }

        // 2. Cache miss, acquire mutex
        String lockKey = LOCK_KEY_PREFIX + productId;
        boolean lockAcquired = redisTemplate.opsForValue()
                .setIfAbsent(lockKey, "1", LOCK_EXPIRE_TIME, TimeUnit.SECONDS);

        if (lockAcquired) {
            try {
                // 3. Double-check cache after acquiring lock
                productJson = redisTemplate.opsForValue().get(cacheKey);
                if (productJson != null) {
                    return JSON.parseObject(productJson, Product.class);
                }

                // 4. Query database
                Product product = productMapper.selectById(productId);
                if (product == null) {
                    // Empty-value cache to prevent penetration
                    redisTemplate.opsForValue()
                            .set(cacheKey, "", 5, TimeUnit.MINUTES);
                    return null;
                }

                // 5. Update cache
                redisTemplate.opsForValue()
                        .set(cacheKey, JSON.toJSONString(product), 
                            CACHE_EXPIRE_TIME, TimeUnit.MINUTES);

                return product;
            } finally {
                // 6. Release lock in finally
                redisTemplate.delete(lockKey);
            }
        } else {
            // 7. Failed to acquire lock, wait 50ms and retry
            try {
                Thread.sleep(50);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
            return getProductById(productId); // recursive retry
        }
    }
}

Technical highlights

Double-checked locking avoids duplicate DB queries

Lock TTL prevents deadlock finally block guarantees lock release

Empty-value cache combined with mutex solves both penetration and breakdown

3. Cache Breakdown: Logical Expiration (High-Performance)

import com.alibaba.fastjson.JSON;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

@Service
public class HotProductService {
    @Autowired
    private StringRedisTemplate redisTemplate;
    @Autowired
    private ProductMapper productMapper;

    private static final String CACHE_KEY_PREFIX = "hot:product:";
    private static final int LOGIC_EXPIRE_TIME = 30 * 60; // logical TTL 30min

    // Thread pool for async cache refresh
    private static final ExecutorService CACHE_REFRESH_EXECUTOR =
        Executors.newFixedThreadPool(10);

    // Cache data object with logical expiration timestamp
    private static class CacheData<T> {
        private T data;
        private Long expireTime; // logical expiration timestamp

        public CacheData(T data, Long expireTime) {
            this.data = data;
            this.expireTime = expireTime;
        }

        // Check if expired
        public boolean isExpired() {
            return System.currentTimeMillis() > expireTime;
        }
    }

    public Product getHotProductById(String productId) {
        String cacheKey = CACHE_KEY_PREFIX + productId;

        // 1. Query cache
        String cacheJson = redisTemplate.opsForValue().get(cacheKey);
        if (cacheJson == null) {
            return null; // hot key pre-warmed, should not reach here
        }

        CacheData<Product> cacheData = JSON.parseObject(cacheJson,
            new TypeReference<CacheData<Product>>() {});
        Product product = cacheData.getData();

        // 2. Check logical expiration
        if (!cacheData.isExpired()) {
            return product; // not expired, return directly
        }

        // 3. Expired: async refresh
        String lockKey = "lock:hot:product:" + productId;
        boolean lockAcquired = redisTemplate.opsForValue()
                .setIfAbsent(lockKey, "1", 10, TimeUnit.SECONDS);

        if (lockAcquired) {
            // Submit async task to update cache
            CACHE_REFRESH_EXECUTOR.submit(() -> {
                try {
                    // Query database
                    Product newProduct = productMapper.selectById(productId);
                    // Update cache with new logical expiration
                    CacheData<Product> newCacheData =
                        new CacheData<>(
                            newProduct,
                            System.currentTimeMillis() + LOGIC_EXPIRE_TIME * 1000L
                        );
                    redisTemplate.opsForValue().set(cacheKey,
                        JSON.toJSONString(newCacheData));
                } finally {
                    redisTemplate.delete(lockKey);
                }
            });
        }

        // 4. Return stale data regardless of lock acquisition
        return product;
    }
}

Technical highlights

Non-blocking: all requests return immediately, no thread waits

Async update: dedicated thread pool handles cache refresh, no impact on main thread

Hot key pre-warming: load hot data into cache at startup to avoid cold-start issues

4. Cache Avalanche: Random TTL Offset

// Add 0-300s random offset to each key's TTL
int baseExpireTime = 30 * 60; // base TTL 30min
int randomOffset = new Random().nextInt(300); // random offset 0-5min
int finalExpireTime = baseExpireTime + randomOffset;

redisTemplate.opsForValue().set(
    cacheKey,
    JSON.toJSONString(product),
    finalExpireTime,
    TimeUnit.SECONDS
);

Technical Challenges & Solutions Summary

Bloom filter false positives : Hash collisions cause non-existent keys to be misjudged as existing. Solution: 1. Lower FPP (more hash functions, larger bit array) 2. Combine with empty-value cache as fallback. Best practice: Set FPP 0.01%–0.1%, rebuild Bloom filter periodically.

Mutex lock deadlock : Service crash leaves lock unreleased. Solution: 1. Set lock TTL 2. Use Redis Lua script for atomic lock release. Best practice: Lock TTL = 2–3× DB query time.

Mutex lock thundering herd : Many threads wait for lock release, then simultaneously hit DB. Solution: 1. Increase retry interval 2. Use striped locks 3. Prefer logical expiration. Best practice: Retry interval 50–100ms to avoid frequent retries.

Logical expiration stale reads : Async update window serves old data. Solution: 1. Shorten logical TTL 2. Use mutex lock for strong consistency on critical paths. Best practice: Non-core (product detail, article) → logical expiration; core (order) → mutex lock.

Redis cluster outage avalanche : Cache layer completely down, all traffic hits DB. Solution: 1. Redis Cluster + Sentinel 2. Local cache fallback 3. Circuit breaker & degradation. Best practice: Local cache with Caffeine, set max size and TTL.

Bulk import avalanche : Mass keys expire simultaneously. Solution: 1. Random TTL offset 2. Batch import 3. Import during low-traffic windows. Best practice: Random offset = 10%–20% of base TTL.

Cache warm-up cold start : Empty cache at startup floods DB. Solution: 1. Pre-load hot data at startup 2. Gradual traffic ramp-up 3. Degradation mechanism. Best practice: Pre-identify hot keys, async load into cache at startup.

Real Interview Simulation

Interviewer 👨‍💻:

“Good basics. Now a scenario design: explain Redis cache penetration, breakdown, avalanche , clearly differentiate them and give solutions. Go ahead.”

Candidate 🧑‍💻:

“All three are essentially cache failing to shield traffic, hammering the database , but triggers differ and so do solutions.

Quick distinction:

🔍 Penetration : Querying data that does not exist in DB at all , cache naturally empty, requests ‘penetrate’ through.

💥 Breakdown : A hot key just expired , massive concurrent requests race to rebuild cache, ‘breaking through’ the DB.

❄️ Avalanche : Mass keys expire in same window , or Redis dies outright, requests avalanche onto DB.

May I walk through each with diagrams?”

Interviewer 👨‍💻:

“Sure, draw and explain.”

Candidate 🧑‍💻:

“First, 🔍 Cache Penetration .

Example: attacker repeatedly requests id = -1, which doesn’t exist in DB nor cache, every request hits DB.

Penetration diagram
Penetration diagram

Two solutions:

Bloom filter : Pre-load all valid IDs. Incoming requests hit filter first; non-existent ones blocked at the gate 🚪.

Cache empty object : Even if DB returns null, store a short-lived placeholder (e.g., "") with 30s TTL. Next identical request caught at cache layer.

In production I use Bloom filter upfront + empty-value fallback , double insurance.”

Interviewer 👨‍💻:

“Clear logic. How about breakdown?”

Candidate 🧑‍💻:

“💥 Cache Breakdown .

Classic case: flash-sale item, key hot_item:123. When this hot key expires during peak traffic, all requests rush to DB to rebuild cache.

Breakdown diagram
Breakdown diagram

Core tactic: prevent concurrent rebuild.

Mutex (Mutex) : Use Redis SETNX for a lightweight lock; only one thread queries DB, others sleep tens of ms and retry cache.

Logical expiration : Hot key has no TTL; value embeds logical expiry timestamp. On logical expiry, return stale value immediately, spawn background thread to refresh. This is the high-performance king 🤖.

Pseudo-code sketch:

public String getHotData(String key) {
    String value = redis.get(key);
    if (value == null) {
        if (redis.setnx(lockKey, "1", 10, TimeUnit.SECONDS)) {
            try {
                value = db.query();
                redis.set(key, value, 3600);
            } finally {
                redis.del(lockKey);
            }
        } else {
            Thread.sleep(50);
            return getHotData(key);
        }
    }
    return value;
}

Interviewer 👨‍💻:

“If using logical expiration, how do you store the value?”

Candidate 🧑‍💻:

“Wrap in an object, e.g., HotDataWrapper, with two fields: data and expireAt (logical expiry timestamp). On read, check expireAt < now; if expired, return old data and spin a thread to acquire lock and rebuild.”

Interviewer 👨‍💻:

“Good. Now avalanche.”

Candidate 🧑‍💻:

“❄️ Cache Avalanche , highest destructive power.

Two overlapping scenarios: mass keys expire together, or Redis crashes.

Avalanche diagram
Avalanche diagram

Multi-pronged defense:

Randomize TTL with salt : Add 1–5 min random offset to base TTL, spreading expiry. Simplest and most effective 😊.

HA architecture : Redis master-replica + Sentinel/Cluster, eliminate cache single point of failure.

Circuit breaker & degradation : Sentinel/Hystrix trips when DB load spikes, returns degraded response (local cache, default, ‘busy’), combined with rate limiting.

Multi-level cache : Local Caffeine + Redis two-tier, Redis down still has local fallback.”

Interviewer 👨‍💻:

“Summarize with a comparison?”

Candidate 🧑‍💻:

“Table summary:

🔍 Penetration: Trigger — query non-existent data; Core idea — Bloom filter / empty object; One-liner — Prevent non-existent

💥 Breakdown: Trigger — hot key expires; Core idea — mutex / logical expiration; One-liner — Prevent rebuild race

❄️ Avalanche: Trigger — mass keys expire / crash; Core idea — randomize TTL / HA / degradation; One-liner — Prevent simultaneous down

Interviewer 👨‍💻:

“Nice. Follow-up: in a clustered deployment, if SETNX lock holder times out or node crashes, deadlock risk? How to solve?”

Candidate 🧑‍💻:

“Real deadlock risk. I add lock TTL and use Redisson’s watchdog auto-renewal — as long as thread lives, lock won’t expire; if node truly dies, lock auto-releases on TTL expiry. No stuck deadlocks.”

Interviewer 👨‍💻:

“Solid. Can you paste core code showing technical highlights, then summarize technical challenges & solutions for this whole scenario?”

Candidate 🧑‍💻:

“Sure, I’ll present by scenario with code, challenges, and insights. 😄”

🔍 Cache Penetration — Core Code

Technical highlights : Bloom filter + empty-value cache dual insurance, balancing performance and penetration defense.

// 1. Bloom filter init (Redisson, at startup)
RBloomFilter<String> bloomFilter = redissonClient.getBloomFilter(
    "userIdFilter");
// Expected 1M inserts, FPP 0.01%
bloomFilter.tryInit(1_000_000L, 0.0001);

// Full ID warm-up, load from DB in batches
List<Long> allUserIds = userMapper.getAllIds();
allUserIds.forEach(id -> bloomFilter.add("user:" + id));

// 2. Query logic
public User getUserById(Long id) {
    String key = "user:" + id;

    // Bloom filter intercept
    if (!bloomFilter.contains(key)) {
        return null; // direct return, no DB hit
    }

    // Check cache
    String cacheValue = redisTemplate.opsForValue().get(key);
    if (cacheValue != null) {
        // Check if empty placeholder
        if ("NULL_PLACEHOLDER".equals(cacheValue)) {
            return null;
        }
        return JSON.parseObject(cacheValue, User.class);
    }

    // Query DB
    User user = userMapper.selectById(id);
    if (user == null) {
        // Cache empty object, 30s TTL to save space
        redisTemplate.opsForValue().set(key, "NULL_PLACEHOLDER", 30, TimeUnit.SECONDS);
    } else {
        redisTemplate.opsForValue().set(key, JSON.toJSONString(user), 1, TimeUnit.HOURS);
    }
    return user;
}

💥 Cache Breakdown — Core Code

Technical highlights : Redisson distributed lock + watchdog auto-renewal , avoids deadlock, ensures high concurrency safety.

public Product getHotProduct(Long id) {
    String key = "hot_product:" + id;
    String cacheData = redisTemplate.opsForValue().get(key);
    if (cacheData != null) {
        return JSON.parseObject(cacheData, Product.class);
    }

    // Distributed lock key
    String lockKey = "lock:product:" + id;
    RLock lock = redissonClient.getLock(lockKey);
    try {
        // Try lock, wait max 10s, lease 30s (watchdog renews, 30s fallback)
        if (lock.tryLock(10, 30, TimeUnit.SECONDS)) {
            // Double-check
            cacheData = redisTemplate.opsForValue().get(key);
            if (cacheData != null) {
                return JSON.parseObject(cacheData, Product.class);
            }
            // Query DB
            Product product = productMapper.selectById(id);
            if (product != null) {
                // Set cache with random TTL (300-420s) to prevent avalanche
                redisTemplate.opsForValue().set(key, JSON.toJSONString(product), 
                    300 + ThreadLocalRandom.current().nextInt(120), TimeUnit.SECONDS);
            } else {
                // Empty-value cache to prevent penetration
                redisTemplate.opsForValue().set(key, "NULL_PLACEHOLDER", 30, TimeUnit.SECONDS);
            }
            return product;
        } else {
            // Lock acquire failed, sleep and retry
            Thread.sleep(50);
            return getHotProduct(id); // recursive retry
        }
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
        return null;
    } finally {
        if (lock.isHeldByCurrentThread()) {
            lock.unlock();
        }
    }
}

❄️ Cache Avalanche — Core Code

Technical highlights : Random TTL salt + multi-level cache fallback (Caffeine).

// Local cache config (Caffeine)
Cache<Long, Object> localCache = Caffeine.newBuilder()
    .maximumSize(1000)
    .expireAfterWrite(60, TimeUnit.SECONDS)
    .build();

public Object getDataWithMultiLevel(Long id) {
    String key = "data:" + id;

    // 1. L1: Local cache
    Object localVal = localCache.getIfPresent(id);
    if (localVal != null) {
        return localVal;
    }

    // 2. L2: Redis
    String cacheVal = redisTemplate.opsForValue().get(key);
    if (cacheVal != null) {
        localCache.put(id, JSON.parse(cacheVal)); // backfill local
        return cacheVal;
    }

    // 3. Query DB with lock (prevent breakdown)
    Object dbVal = queryFromDbWithLock(id);
    if (dbVal != null) {
        // Set Redis TTL: base 1h + random 0-600s
        long expireSec = 3600 + ThreadLocalRandom.current().nextInt(600);
        redisTemplate.opsForValue().set(key, JSON.toJSONString(dbVal), expireSec, TimeUnit.SECONDS);
        localCache.put(id, dbVal);
    }
    return dbVal;
}

🧩 Scenario Technical Challenges & Solutions

🔍 Penetration : Challenges — 1. Bloom filter cannot delete , requires rebuild on data changes. 2. False positives let some invalid requests through. 3. Empty-value cache consumes memory, may be abused with crafted keys. Solutions — 1. Use scalable Bloom filter (RedisBloom module) or scheduled rebuild. 2. Set FPP ≤ 0.01%, rely on empty-value cache as safety net. 3. Empty-value cache with ultra-short TTL, plus request rate limiting. Experience tip: Actual FPP must be stress-tested, not just theoretical.

💥 Breakdown : Challenges — 1. Lock granularity vs performance : more hot keys = heavier lock contention. 2. Logical expiration has eventual consistency — async update may return stale data. 3. Distributed lock deadlock risk . Solutions — 1. Single-node: ConcurrentHashMap + lightweight lock; cluster: Redisson fine-grained locks. 2. Ensure stale value remains usable, pre-load or background refresh. 3. Redisson watchdog auto-renewal + finally unlock. Experience tip: Logical expiration suits read-heavy, write-light workloads tolerating brief inconsistency.

❄️ Avalanche : Challenges — 1. Random TTL algorithm can still produce local hot-spot clusters. 2. Multi-level cache sync issues — local vs Redis may diverge. 3. Post-circuit-breaker recovery causes ‘pulse’ pressure spikes. Solutions — 1. Combine randomness with business-aware pre-warming of high-frequency keys. 2. Short local TTL, message queue for invalidation. 3. Half-open state gradual ramp-up via token bucket or warm-up algorithm. Experience tip: Production must have multi-level cache; core interfaces need stress-tested QPS limits and thresholds. 👍

Interviewer 👨‍💻:

“Excellent, code details spot-on, challenges broken down clearly. Your habit of linking code to solutions is a strong interview signal. This cache trio assessment earns you a Strong Hire ✅.”

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

JavaRedishigh concurrencyCaffeinedistributed lockinterview preparationBloom filterRedissoncache avalanchecache breakdowncache penetrationmulti-level cachinglogical expiration
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.