How to Stop Redis Cache Penetration, Breakdown, and Avalanche – Proven Solutions Inside

Overview

In high‑concurrency internet applications, Redis is a critical cache layer. Three major cache problems—cache penetration, cache breakdown, and cache avalanche—can cause massive database load or system collapse if not handled properly.

Cache penetration occurs when queries request data that does not exist, causing every request to bypass the cache and hit the database. Cache breakdown happens when a hot key expires and a flood of concurrent requests hit the database. Cache avalanche is the simultaneous expiration of many keys, directing all traffic to the database.

This article analyzes the nature of these problems and provides production‑tested solutions such as Bloom filters, distributed locks, never‑expire hot data, and cache warming.

Technical Characteristics

Cache Penetration Protection : Bloom filter, empty‑value caching.

Cache Breakdown Protection : Mutex lock, logical expiration.

Cache Avalanche Prevention : Random expiration, cache pre‑warming, multi‑level cache.

High Availability : Redis cluster, Sentinel, persistence.

Performance Optimization : Proper cache strategy can reduce DB queries by over 90% and bring response time to milliseconds.

Monitoring & Alerts : Real‑time monitoring of hit rate, slow queries, and key metrics.

Applicable Scenarios

E‑commerce flash‑sale (hot product inventory).

Social platforms (user profiles, hot posts).

News sites (burst traffic for breaking news).

Financial systems (account balance, transaction records).

Content delivery (static assets via CDN).

API gateways (rate limiting, permission checks).

Environment Requirements

Component

Version

Notes

OS

CentOS 7+/Ubuntu 20.04+

Supported Linux distributions

Redis

6.0+

Recommend latest stable version with ACL support

Java

JDK 1.8+

Spring Boot example

Spring Boot

2.5+

Integrated Redis client and distributed lock

Redisson

3.16+

Provides advanced features

Hardware

4‑core CPU, 8 GB RAM+

Higher specs recommended for production

Network

≥100 Mbps

Low‑latency between Redis and app servers

Detailed Steps

1. Preparation

1.1 System Check

# Check OS version
cat /etc/os-release

# Check resources
free -h
df -h

# Check network
ping -c 3 redis.example.com

# Check port usage
netstat -tulnp | grep 6379

# Check time sync
timedatectl status

1.2 Install Redis

# Ubuntu/Debian
sudo apt update
sudo apt install -y redis-server redis-tools

# CentOS/RHEL
sudo yum install -y epel-release
sudo yum install -y redis

# Build from source (recommended)
wget https://download.redis.io/releases/redis-7.0.15.tar.gz
tar -xzf redis-7.0.15.tar.gz
cd redis-7.0.15
make
sudo make install

# Verify installation
redis-server --version
redis-cli --version

1.3 Basic Redis Configuration

# Edit /etc/redis/redis.conf
# bind 0.0.0.0                # Listen on all interfaces
# protected-mode yes          # Enable protection mode
# requirepass your_strong_password
# maxmemory 4gb
# maxmemory-policy allkeys-lru
# appendonly yes
# appendfsync everysec
# systemctl start redis
# systemctl enable redis
# redis-cli -h localhost -p 6379 -a your_password ping

Core Configurations

2.1 Cache Penetration Solutions

Solution 1: Bloom Filter

Bloom filter is a space‑efficient probabilistic data structure that quickly determines whether a key may exist.

// Maven dependency
// <dependency>
//   <groupId>com.google.guava</groupId>
//   <artifactId>guava</artifactId>
//   <version>31.1-jre</version>
// </dependency>

import com.google.common.hash.BloomFilter;
import com.google.common.hash.Funnels;
import org.springframework.stereotype.Component;

@Component
public class BloomFilterService {
    private static final int EXPECTED_INSERTIONS = 10000000;
    private static final double FPP = 0.001;
    private BloomFilter<Long> bloomFilter;

    public BloomFilterService() {
        bloomFilter = BloomFilter.create(Funnels.longFunnel(), EXPECTED_INSERTIONS, FPP);
        initBloomFilter();
    }

    /** Load all valid IDs from DB */
    private void initBloomFilter() {
        List<Long> allIds = userMapper.selectAllIds();
        allIds.forEach(bloomFilter::put);
        System.out.println("Bloom filter initialized, loaded " + allIds.size() + " entries");
    }

    public boolean mightContain(Long id) {
        return bloomFilter.mightContain(id);
    }

    public void put(Long id) {
        bloomFilter.put(id);
    }
}

Solution 2: Empty‑Value Caching

@Service
public class UserService {
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    @Autowired
    private UserMapper userMapper;
    @Autowired
    private BloomFilterService bloomFilterService;

    public User getUserById(Long userId) {
        String key = "user:" + userId;
        // Bloom filter check
        if (!bloomFilterService.mightContain(userId)) {
            log.warn("Bloom filter blocked illegal request: userId={}", userId);
            return null;
        }
        // Cache lookup
        User user = (User) redisTemplate.opsForValue().get(key);
        if (user != null) {
            if (user.getId() == null) {
                return null; // empty placeholder
            }
            return user;
        }
        // DB fallback
        user = userMapper.selectById(userId);
        if (user != null) {
            redisTemplate.opsForValue().set(key, user, 30, TimeUnit.MINUTES);
            return user;
        } else {
            // Cache empty placeholder for 5 minutes
            redisTemplate.opsForValue().set(key, new User(), 5, TimeUnit.MINUTES);
            return null;
        }
    }
}

2.2 Cache Breakdown Solutions

Solution 1: Mutex Lock

@Service
public class ProductService {
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    @Autowired
    private RedissonClient redissonClient;
    @Autowired
    private ProductMapper productMapper;

    public Product getProductById(Long productId) {
        String key = "product:" + productId;
        String lockKey = "lock:product:" + productId;
        Product product = (Product) redisTemplate.opsForValue().get(key);
        if (product != null) {
            return product;
        }
        RLock lock = redissonClient.getLock(lockKey);
        try {
            if (lock.tryLock(10, 30, TimeUnit.SECONDS)) {
                // Double‑check after acquiring lock
                product = (Product) redisTemplate.opsForValue().get(key);
                if (product != null) {
                    return product;
                }
                product = productMapper.selectById(productId);
                if (product != null) {
                    redisTemplate.opsForValue().set(key, product, 30, TimeUnit.MINUTES);
                }
                return product;
            } else {
                Thread.sleep(50);
                return getProductById(productId);
            }
        } catch (InterruptedException e) {
            log.error("Failed to acquire lock", e);
            throw new RuntimeException("System busy, please retry");
        } finally {
            if (lock.isHeldByCurrentThread()) {
                lock.unlock();
            }
        }
    }
}

Solution 2: Logical Expiration (Recommended for Hot Data)

@Data
public class RedisData<T> {
    private LocalDateTime expireTime; // logical expiration
    private T data;
}

@Service
public class HotProductService {
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    @Autowired
    private RedissonClient redissonClient;
    @Autowired
    private ProductMapper productMapper;
    private static final ExecutorService CACHE_REBUILD_EXECUTOR = Executors.newFixedThreadPool(10);

    public Product getHotProduct(Long productId) {
        String key = "hot:product:" + productId;
        RedisData<Product> redisData = (RedisData<Product>) redisTemplate.opsForValue().get(key);
        if (redisData == null) {
            return getFromDBAndCache(productId);
        }
        if (redisData.getExpireTime().isAfter(LocalDateTime.now())) {
            return redisData.getData();
        }
        // Asynchronously rebuild cache
        String lockKey = "lock:rebuild:" + productId;
        RLock lock = redissonClient.getLock(lockKey);
        try {
            if (lock.tryLock(0, 10, TimeUnit.SECONDS)) {
                CACHE_REBUILD_EXECUTOR.submit(() -> {
                    try {
                        rebuildCache(productId);
                    } finally {
                        lock.unlock();
                    }
                });
            }
        } catch (InterruptedException e) {
            log.error("Failed to acquire lock", e);
        }
        return redisData.getData(); // return stale data
    }

    private Product getFromDBAndCache(Long productId) {
        Product product = productMapper.selectById(productId);
        if (product != null) {
            RedisData<Product> data = new RedisData<>();
            data.setData(product);
            data.setExpireTime(LocalDateTime.now().plusHours(1));
            redisTemplate.opsForValue().set("product:" + productId, data);
        } else {
            redisTemplate.opsForValue().set("product:" + productId, new RedisData<>(), 5, TimeUnit.MINUTES);
        }
        return product;
    }

    private void rebuildCache(Long productId) {
        Product product = productMapper.selectById(productId);
        if (product != null) {
            RedisData<Product> data = new RedisData<>();
            data.setData(product);
            data.setExpireTime(LocalDateTime.now().plusHours(1));
            redisTemplate.opsForValue().set("product:" + productId, data);
            log.info("Product cache rebuilt: productId={}", productId);
        }
    }
}

2.3 Cache Avalanche Solutions

Solution 1: Random Expiration

@Service
public class CacheService {
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    private static final Random RANDOM = new Random();

    public void setWithRandomExpire(String key, Object value, long baseExpire, TimeUnit unit) {
        long randomSeconds = RANDOM.nextInt(300);
        long finalExpire = unit.toSeconds(baseExpire) + randomSeconds;
        redisTemplate.opsForValue().set(key, value, finalExpire, TimeUnit.SECONDS);
    }

    public void batchSetWithRandomExpire(Map<String, Object> dataMap, long baseExpire, TimeUnit unit) {
        dataMap.forEach((k, v) -> setWithRandomExpire(k, v, baseExpire, unit));
    }
}

Solution 2: Redis Cluster + Multi‑Level Cache

@Configuration
public class CacheConfig {
    @Bean
    public CacheManager caffeineCacheManager() {
        CaffeineCacheManager manager = new CaffeineCacheManager();
        manager.setCaffeine(Caffeine.newBuilder()
                .initialCapacity(100)
                .maximumSize(1000)
                .expireAfterWrite(5, TimeUnit.MINUTES)
                .recordStats());
        return manager;
    }

    @Bean
    public CacheManager redisCacheManager(RedisConnectionFactory factory) {
        RedisCacheConfiguration config = RedisCacheConfiguration.defaultCacheConfig()
                .entryTtl(Duration.ofMinutes(30))
                .serializeKeysWith(RedisSerializationContext.SerializationPair.fromSerializer(new StringRedisSerializer()))
                .serializeValuesWith(RedisSerializationContext.SerializationPair.fromSerializer(new GenericJackson2JsonRedisSerializer()))
                .disableCachingNullValues();
        return RedisCacheManager.builder(factory).cacheDefaults(config).build();
    }
}

@Service
public class MultiLevelCacheService {
    @Autowired
    @Qualifier("caffeineCacheManager")
    private CacheManager localCacheManager;
    @Autowired
    @Qualifier("redisCacheManager")
    private CacheManager redisCacheManager;
    @Autowired
    private UserMapper userMapper;

    public User getUser(Long userId) {
        String key = String.valueOf(userId);
        Cache localCache = localCacheManager.getCache("users");
        User user = localCache.get(key, User.class);
        if (user != null) {
            log.debug("Hit local cache: userId={}", userId);
            return user;
        }
        Cache redisCache = redisCacheManager.getCache("users");
        user = redisCache.get(key, User.class);
        if (user != null) {
            log.debug("Hit Redis cache: userId={}", userId);
            localCache.put(key, user);
            return user;
        }
        user = userMapper.selectById(userId);
        if (user != null) {
            redisCache.put(key, user);
            localCache.put(key, user);
        }
        return user;
    }
}

Best Practices and Precautions

Performance Optimization

Set appropriate expiration times : hot data 1‑2 h, normal data 30 min, cold data 5 min, statistics 1 day.

Use Pipeline for batch operations to reduce network round‑trips.

Leverage Redis data structures such as Hash for objects and Bitmap for bitmaps.

Security Hardening

Enable strong password authentication.

Restrict access with firewall rules (e.g., allow only application servers).

Rename or disable dangerous commands (FLUSHDB, KEYS, CONFIG).

Enable SSL/TLS for client connections.

High Availability

Configure master‑slave replication.

Deploy Sentinel for automatic failover.

Use Redis Cluster for sharding and fault tolerance.

Troubleshooting and Monitoring

Common Issues

Symptom

Cause

Solution

Cache avalanche, DB overload

Simultaneous key expiration

Random expiration, cache pre‑warming, multi‑level cache

Cache breakdown, DB spike

Hot key expires

Mutex lock, logical expiration, never‑expire hot data

Cache penetration attacks

Requests for nonexistent data

Bloom filter, empty‑value cache, rate limiting

High Redis memory usage

Too many keys or long TTL

Set maxmemory, choose proper eviction policy, clean stale data

Data inconsistency

Improper update order, lock loss

Delayed double delete, MQ for eventual consistency

Distributed lock deadlock

Service crash while holding lock

Set lock auto‑expire, use Redisson watchdog, always release in finally

Performance Monitoring

Key metrics: CPU, memory, hit rate, QPS, connection count, latency, slow queries, replication lag.

Use Prometheus + Grafana with redis_exporter. Example alert rules for memory usage, low hit rate, service down, replication lag are provided.

Backup and Recovery

Backup Script

#!/bin/bash
REDIS_CLI="/usr/bin/redis-cli"
REDIS_PASSWORD="your_password"
BACKUP_DIR="/data/redis_backup"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=7

mkdir -p "${BACKUP_DIR}"
${REDIS_CLI} -a "${REDIS_PASSWORD}" --no-auth-warning BGSAVE
while [ $(${REDIS_CLI} -a "${REDIS_PASSWORD}" --no-auth-warning LASTSAVE) -ne $(${REDIS_CLI} -a "${REDIS_PASSWORD}" --no-auth-warning LASTSAVE) ]; do sleep 1; done
cp /var/lib/redis/dump.rdb "${BACKUP_DIR}/dump_${DATE}.rdb"
gzip "${BACKUP_DIR}/dump_${DATE}.rdb"
if [ -f /var/lib/redis/appendonly.aof ]; then
    cp /var/lib/redis/appendonly.aof "${BACKUP_DIR}/appendonly_${DATE}.aof"
    gzip "${BACKUP_DIR}/appendonly_${DATE}.aof"
fi
find "${BACKUP_DIR}" -name "*.gz" -mtime +${RETENTION_DAYS} -delete
echo "Backup completed: ${DATE}"

Recovery Procedure

Stop Redis service.

Decompress the desired RDB (or AOF) file and replace /var/lib/redis/dump.rdb.

Verify integrity with redis-check-rdb.

Start Redis and confirm data with redis-cli DBSIZE and sample GET/SET.

Conclusion

This guide covers the root causes of Redis cache penetration, breakdown, and avalanche, and presents proven mitigation techniques—including Bloom filters, distributed locks, logical expiration, random TTL, cache pre‑warming, multi‑level caching, and high‑availability deployment. Proper monitoring, alerting, and backup strategies complete a robust production‑grade caching architecture.