Prevent Cache Avalanche with Multi‑Level Caffeine + Redis: High‑Availability Design

Why Cache Is Critical in High‑Concurrency Distributed Systems

Relying solely on a Redis cache layer is risky because cluster jitter, mass key expiration, or node failures can trigger a cache avalanche, causing massive request bursts to hit the database, saturating CPU and connection pools, and rendering the service unavailable.

How Cache Avalanche Happens

A cache avalanche occurs when a large number of requests bypass the cache and hit the database simultaneously. Common causes include:

Mass key expiration at the same timestamp – all keys become invalid at once, flooding the database.

Redis cluster outage – master‑slave switch‑over stalls, network partitions, or shard failures.

Poor cache warm‑up after restart – old cache is cleared while new cache is not yet loaded, creating a vacuum period.

Memory eviction triggering bulk eviction – when Redis memory is full, LRU eviction removes many keys at once.

It is important to differentiate cache breakdown (single hot key expires) from cache avalanche (mass key expiration or whole‑cluster failure). The former and latter require different mitigation strategies.

Multi‑Level Cache Architecture

The proposed three‑layer design is:

L1 Local Cache – Caffeine; stores high‑frequency hot data (e.g., homepage top products, activity configs, logged‑in user info) with a TTL of 5‑10 minutes plus random offset; zero network overhead; acts as the first line of defense.

L2 Distributed Cache – Redis cluster; stores the full business dataset with hot keys never expiring and cold keys having short TTLs plus random offset; ensures data consistency across the cluster.

L3 Data Source – MySQL; holds the ultimate source of truth and serves as the final fallback.

Core Request Flow

Requests follow a "near‑to‑far" interception principle: first check the local Caffeine cache, then Redis, and finally the database. In practice, the local cache intercepts over 90% of hotspot traffic, the Redis layer handles the remaining medium‑frequency requests, and the database only processes rare cache misses.

Key Strategies to Prevent Cache Avalanche

1. Randomized Expiration Times

Never use a fixed TTL for non‑permanent keys. Add a random offset to the base TTL to spread expirations over time. Example implementation:

// Base TTL 300 seconds, random offset 0‑120 seconds
long baseTtl = 300;
long randomOffset = ThreadLocalRandom.current().nextInt(120);
long finalTtl = baseTtl + randomOffset;
redisTemplate.opsForValue().set(key, value, finalTtl, TimeUnit.SECONDS);

2. Separate Hot and Cold Data into Different Redis Clusters

Hot cluster – stores core hot data on high‑spec machines; keys never expire; refreshed proactively via Canal listening to MySQL binlog.

Cold cluster – stores low‑frequency data (historical orders, old activities) on lightweight nodes; allows random eviction.

This isolation prevents bulk eviction of cold data from crowding out hot data memory.

3. Never Expire Hot Keys

For critical data such as flash‑sale items or homepage recommendations, do not set an expiration in Redis. Updates are propagated by listening to MySQL binlog and asynchronously refreshing Redis, while also pushing the latest value to the local Caffeine cache.

4. Local Cache Fallback When Redis Fails

If Redis experiences timeouts, node crashes, or master‑slave switch‑overs, the application catches the exception and degrades directly to the local Caffeine cache, avoiding further database queries. After Redis recovers, data is synchronized via version numbers or scheduled tasks.

public Object getData(String key) {
    // 1. Check local cache
    Object val = caffeineCache.getIfPresent(key);
    if (val != null) return val;
    try {
        // 2. Query Redis with a short timeout
        Object redisVal = redisTemplate.opsForValue().get(key, 200, TimeUnit.MILLISECONDS);
        if (redisVal != null) {
            caffeineCache.put(key, redisVal);
            return redisVal;
        }
    } catch (Exception e) {
        // 3. Redis failure – fallback to local cache if present
        Object fallback = caffeineCache.getIfPresent(key);
        if (fallback != null) {
            log.warn("Redis unavailable, using local fallback for key: {}", key);
            return fallback;
        }
        // 4. If nothing is cached, consider rate‑limiting before DB access
    }
    // ... DB access logic ...
}

5. Rate‑Limiting and Circuit‑Breaker as the Last Guard

In extreme cases where all caches miss, a rate‑limiting circuit‑breaker (e.g., Sentinel or Resilience4j) is placed at the database access layer. When QPS exceeds a configured threshold, the circuit opens and returns predefined fallback data (e.g., empty objects or stale cache entries) to protect the database.

Caffeine Local Cache Configuration Details

@Configuration
public class CaffeineConfig {
    @Bean
    public Cache<String, Object> caffeineCache() {
        return Caffeine.newBuilder()
            // Max entries, typically 5‑10% of JVM heap, not exceeding 2GB
            .maximumSize(10_000)
            // Expire 5 minutes after write
            .expireAfterWrite(5, TimeUnit.MINUTES)
            // Expire 3 minutes after access to avoid cold data staying too long
            .expireAfterAccess(3, TimeUnit.MINUTES)
            // Enable statistics for monitoring hit rate
            .recordStats()
            // Removal listener for logging/monitoring
            .removalListener((key, value, cause) ->
                log.debug("Cache removed: {}, cause: {}", key, cause))
            .build();
    }
}

Memory planning advice: for a 4 GB heap, keep the local cache size around 200‑400 MB (≈10‑20 k objects). Oversizing can cause frequent GC pauses. Expose caffeineCache.stats() to monitoring platforms (e.g., Prometheus) and adjust size based on hit‑rate.

Performance Test Comparisons

Test Scenario                | Architecture   | DB Peak QPS | Avg Resp Time | Service Availability
-----------------------------|----------------|--------------|----------------|----------------------
Redis 30‑s cluster jitter    | Single Redis   | ~50k (full fallback) | 800ms+        | ~80%
Redis 30‑s cluster jitter    | Multi‑Level    | 400‑600      | 7‑9ms          | 99.99%
Mass key expiration         | Single Redis   | 35k‑40k      | 600ms+        | 87%
Mass key expiration         | Multi‑Level    | 300‑500      | ~8ms           | 99.98%

Note: The numbers come from an internal benchmark environment (10 application nodes × 8 C/16 G, Redis cluster with 8 shards). Exact values may vary with business models and hardware, but the trends are representative.

Cache Consistency Handling

6.1 Strong Consistency Scenarios (e.g., payment, inventory, account balance)

These scenarios cannot tolerate dirty data. After updating the database, the Redis entry is deleted, and an MQ broadcast clears the local Caffeine cache on all nodes.

@Transactional
public void updateProduct(Product product) {
    // 1. Update DB
    productMapper.update(product);
    // 2. Delete Redis cache
    redisTemplate.delete("product:" + product.getId());
    // 3. Send MQ message to clear local cache
    mqTemplate.send("cache-clear-topic", "product:" + product.getId());
}

@MqListener(topic = "cache-clear-topic")
public void handleCacheClear(String key) {
    caffeineCache.invalidate(key);
}

6.2 Eventual Consistency Scenarios (e.g., product details, news content)

Use a delayed double‑delete strategy: after DB update, delete Redis immediately, then delete it again after a delay (typically MySQL master‑slave sync latency plus average write time) to catch any stale writes.

public void updateArticle(Article article) {
    articleMapper.update(article);
    String key = "article:" + article.getId();
    redisTemplate.delete(key);
    // Delayed second delete
    scheduler.schedule(() -> {
        redisTemplate.delete(key);
        // Also broadcast to clear local cache
    }, getDelayMillis(), TimeUnit.MILLISECONDS);
}

6.3 Periodic Refresh as a Safety Net

Critical hot data is fully refreshed every 5 minutes from the DB into both Redis and the local cache, preventing long‑term stale data during abnormal conditions.

Common Pitfalls and Solutions

Pitfall 1: Local Cache Polluted by Cold Data

Cold keys loaded into the limited‑size local cache can evict true hot data. Solutions:

Set an appropriate maximumSize based on business characteristics.

Monitor stats().hitRate(); if hit‑rate falls below 80 %, investigate cold‑key influx.

Only write keys accessed more than a threshold within the last 5 minutes into the local cache.

Pitfall 2: Stale Local Cache After Redis Recovery

During degradation, the local cache may serve outdated data. Introduce a version field in Redis values; each DB update increments the version. Local cache stores the same version. An asynchronous task compares versions and refreshes the local entry when Redis holds a newer version.

// Data structure
public class CacheValue<T> {
    private T data;
    private Long version; // incremented on each DB update
}

public Object getWithVersion(String key) {
    CacheValue local = (CacheValue) caffeineCache.getIfPresent(key);
    asyncExecutor.submit(() -> {
        CacheValue redisVal = (CacheValue) redisTemplate.opsForValue().get(key);
        if (redisVal != null && local != null && redisVal.getVersion() > local.getVersion()) {
            caffeineCache.put(key, redisVal);
        }
    });
    return local != null ? local.getData() : loadFromRedis(key);
}

Conclusion

In summary, the Caffeine + Redis multi‑level cache architecture boils down to four essential practices:

Local cache intercepts hot traffic – shields Redis from full‑scale load during jitter.

Separate hot and cold data – distinct expiration policies prevent mutual eviction.

Randomize expiration times – eliminates the root cause of bulk key expiration.

Provide Redis fallback with local cache and rate‑limiting circuit‑breaker – blocks abnormal traffic from overwhelming the database.

By following these guidelines and avoiding a one‑size‑fits‑all TTL strategy, systems can achieve robust high‑availability under peak loads.