Redis Production Fault Troubleshooting: 8 Critical Issues & Solutions

In previous articles, we thoroughly covered server, JVM, thread, memory, and slow SQL full-chain application-layer faults, completely mastering Java application underlying troubleshooting capabilities.

Starting with this article, we officially enter Module Three: Middleware Production Fault Troubleshooting.

In production microservice architectures, Redis is the absolute core hub, with cache carrying over half of online traffic. Simultaneously, Redis is also the middleware with the highest incidence of production incidents.

This article comprehensively covers the eight most frequent Redis production faults: cache penetration, cache breakdown, cache avalanche, big keys, hot keys, connection exhaustion, memory overflow, and cache data consistency issues. It includes a full troubleshooting SOP, implementation solutions, and architectural optimizations directly applicable to production and interviews.

1. Redis Core Production Fault Panorama

All Redis production issues fall within the following eight faults; all bizarre timeouts, jitters, and avalanches can be categorized accordingly:

Cache Penetration : Querying non-existent data bypasses cache and hits database directly.

Cache Breakdown : Hot key expires instantly, massive traffic hits database directly.

Cache Avalanche : Large number of keys expire simultaneously or Redis crashes, overall traffic avalanches to DB.

Big Key Fault : Single key data too large, causing network card saturation, Redis stalls, cluster skew.

Hot Key Fault : Single key bears ultra-high QPS, single machine overwhelmed, cluster fails.

Connection Exhaustion : Client connection leaks, unreasonable pool configuration, unable to create new connections.

Memory OOM : Cache without eviction or expiration, memory continuously expands causing Redis crash.

Cache Consistency Issue : Cache and database data inconsistent, triggering business data anomalies.

Each will be broken down: symptoms, root cause, troubleshooting, stop-gap, root cure, architectural optimization — full implementation.

2. Cache Penetration: Traffic Mysteriously Saturates Database

1. Fault Symptoms

Database QPS inexplicably spikes under heavy pressure, Redis hit rate extremely low, large number of requests penetrate directly to DB. Queried data are mostly non-existent invalid data .

2. Core Root Cause

Clients query large amounts of non-existent IDs, malicious order brushing, crawler attacks; cache misses, database misses, every request goes to DB query and never writes to cache , causing permanent penetration.

3. Solutions (Production Standard)

Empty Value Caching : Cache empty/default values for null queries with short TTL to intercept repeated penetration.

Bloom Filter : Pre-filter hot data with Bloom filter; non-existent data intercepted directly without accessing Redis/DB.

API Parameter Validation : Intercept illegal IDs, negative IDs, out-of-bound parameters to eliminate invalid queries.

3. Cache Breakdown: Hot Key Expiration Instant Avalanche

1. Fault Symptoms

Business normally stable, at a certain moment DB pressure surges, interfaces batch timeout , auto-recovers after seconds, recurs periodically.

2. Core Root Cause

Ultra-high concurrency hot key expires ; at expiration moment tens of thousands QPS simultaneously penetrate to database, instantly overwhelming DB.

3. Root Cure Solutions

Mutex Lock (Distributed Lock) : At expiration only allow one request to query DB and rewrite cache; others wait.

Hot Key Never Expires : For ultra-hot operational/config data, remove expiration, update asynchronously in background.

Randomize Expiration Time : Avoid batch hot keys expiring simultaneously.

4. Cache Avalanche: Mass Keys Expire Simultaneously

1. Fault Symptoms

At exact hour/scheduled time, all business interfaces timeout en masse, database CPU maxed out, service avalanche , impact extremely wide.

2. Core Root Cause

Large batch of cache keys share same expiration time, collectively invalidate at same moment, all traffic pours into database.

Another avalanche scenario: Redis cluster crash, master-slave switch, network jitter , cache entirely unavailable.

3. Implementation Solutions

Random Expiration Jitter : Add random value to unified TTL to disperse expiration peaks.

Multi-level Cache Architecture : Local Caffeine cache + Redis distributed cache, dual-layer safety net.

Service Circuit Breaker Degradation : Auto-degrade on Redis anomaly, intercept traffic to protect DB.

Redis High Availability Architecture : Master-slave + Sentinel / Cluster, eliminate single point of failure.

5. Big Key Fault: Redis Stalls, Network Card Saturated, Cluster Skew

Big keys are the most common, most hidden, most performance-destroying problem in production Redis.

1. Fault Symptoms

Redis CPU spikes, response stalls, latency increases.

Interfaces occasionally timeout, pipeline batch operations block.

Cluster node memory uneven, data skew, load imbalance.

Network card traffic instantly saturated, triggering global jitter.

2. Big Key Judgment Criteria (Production General)

String type: value > 10KB

List/Hash/Set/ZSet: element count > 1000

3. Big Key Hazards

Redis single-threaded model; operating on big keys blocks main thread , all commands queue waiting, global stall. Simultaneously triggers network IO surge, cluster migration failures.

4. Troubleshooting and Optimization

Troubleshooting command:

# Scan big keys
redis-cli --bigkeys

Root cure solutions:

Big Key Splitting : Split large List into multiple keys, shard storage.

Prohibit Full Reads : Use hscan, sscan for paginated traversal; forbid one-time full data fetch.

Clean Invalid Data : Delete expired zombie data promptly to avoid continuous key bloat.

6. Hot Key Fault: Single Point Overwhelmed, Cluster Failure

1. Fault Symptoms

One Redis node QPS extremely high, CPU saturated, other nodes idle, cluster thoroughly load-imbalanced , hot node crushed.

2. Root Cause

Flash sale items, homepage configs, event data — hot keys concentrate massive QPS on same Redis slot, same node.

3. Solutions

Local Cache Fallback : Hot keys prioritized in local cache, bypass Redis network IO.

Hot Key Multi-replica Dispersion : key + random suffix to disperse slots, distribute node pressure.

Request Rate Limiting : Pre-limit hot interfaces to protect middleware.

7. Redis Connection Exhaustion Fault

1. Symptoms

Service suddenly cannot connect to Redis, error: Too many connections , interfaces timeout en masse.

2. Root Causes

Client connection pool configured too small.

Connections not released, connection leaks.

Frequent short connections creation exhausts maxclients.

3. Troubleshooting and Fix

# View current connections
redis-cli info clients

Optimization solutions:

Properly configure Lettuce/Jedis connection pool parameters.

Set connection timeout, idle recycling to eliminate connection leaks.

Increase Redis max connections limit.

8. Redis Memory Full and Eviction Policy Fault

1. Symptoms

Redis memory continuously rises, not released, memory OOM, write failures, cache eviction anomalies.

2. Root Cause

No memory eviction policy configured, large number of keys without expiration, cold data piles up indefinitely.

3. Production Optimal Eviction Policy

Online universal standard: allkeys-lru

Prioritizes evicting least recently used keys, fits vast majority of business scenarios, prevents unlimited memory growth.

9. Cache and Database Consistency Issue

1. Fault Symptoms

Database data updated but cache data stale, causing user query inconsistency, business bugs.

2. Optimal Update Strategy (Production Implementation)

Update database first, then delete cache

Combined with delayed double delete mechanism to resolve transient inconsistency from concurrent updates, balancing performance and data accuracy.

10. Redis Production Fault Universal Troubleshooting SOP

For online Redis anomalies, directly apply this process to quickly locate root cause:

Check Redis monitoring: CPU, memory, QPS, hit rate, connection count.

Investigate presence of big keys, hot keys.

Verify cache expiration policy, eviction policy.

Examine client connection pool, timeouts, leak status.

Analyze traffic characteristics of penetration, breakdown, avalanche.

Emergency degradation stop-gap, optimize architecture, complete prevention mechanisms.

11. Article Summary

This article officially opens the middleware troubleshooting chapter, fully covering Redis online nine core faults , thoroughly solving cache avalanche, penetration, breakdown, big key hot key, connection exhaustion, memory overflow, data consistency and other production stubborn issues.

Cache problems are the core watershed distinguishing CRUD programmers from high-availability architecture engineers , and are must-know topics for interviews and production incident postmortems.

12. Next Episode Preview

Next: MQ Message Queue Production Fault Full Troubleshooting, deep dive into RabbitMQ/RocketMQ/Kafka message loss, duplication, backlog, consumption stalls, dead letter queues, cluster faults and other high-frequency issues, completing the last core piece of middleware troubleshooting.