How a Full Redis Connection Pool Triggered a Service Outage: Step‑by‑Step Investigation

Problem Background

During a peak hour (10:00 AM) the platform generated alerts: page load failures, homework submission timeouts, and Java applications throwing

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

. Redis monitoring showed connected_clients hitting the configured maxclients=10000 and rejected_connections appearing. The saturated connections caused authentication, session and cache services to fail, leading to a small‑scale business avalanche.

Redis Connection Mechanism

How Redis Handles Connections

Redis uses a single‑threaded event loop (network I/O can be multithreaded from 6.0, but core processing remains single‑threaded). Each client occupies a file descriptor (fd) and events are processed via epoll or kqueue. Key configuration items:

maxclients 10000   # default maximum connections
timeout 0          # idle timeout (0 = never)
tcp-keepalive 300   # TCP keepalive interval

Impact of Reaching maxclients

When maxclients is reached Redis stops accepting new connections and logs:

# Redis 9001  refused connection (maxclients)
-ERR max number of clients reached

All services that depend on Redis are affected:

New requests cannot obtain a Redis connection → business threads block.

Blocked threads retry, filling the thread pool → web container runs out of threads.

Health checks fail → nodes are removed from the service registry.

Traffic shifts to remaining nodes → their Redis connections also surge → cascading failure.

Investigation Process

Directly Check Redis Connection Status

$ redis-cli -h <redis_host> -p 6379 -a <password> INFO clients
connected_clients:10000
maxclients:10000

Both connected_clients and maxclients are at the limit.

Examine Connection Distribution

$ redis-cli CLIENT LIST

Important fields: addr: client IP and port age: connection lifetime (seconds) idle: idle time (seconds) flags: N=normal, M=master, S=slave

Count connections per IP:

$ redis-cli CLIENT LIST | awk '{print $2}' | cut -d= -f2 | cut -d: -f1 | sort | uniq -c | sort -rn | head -10
   3000 10.0.1.12
   2800 10.0.1.14
   1500 10.0.1.13
   1200 10.0.1.15

Find Zombie Connections

# Find connections idle > 300 s
$ redis-cli CLIENT LIST | awk -F'[ =]' '{for(i=1;i<=NF;i++) if($i=="idle") print $(i+1), $0}' | awk '$1>300' | head -20

Many connections have idle values between 600 s and 3600 s, indicating long‑idle zombie connections. The timeout is set to 0 (never expires).

Verify System‑Level Limits

# Redis process fd limit
$ cat /proc/$(pidof redis-server)/limits | grep "Max open files"
Max open files            10024            10024            files
# System fd limit
$ cat /proc/sys/fs/file-max
100000
# Current fd usage
$ cat /proc/sys/fs/file-nr
30000   0   100000

Redis's Max open files (10024) is close to maxclients (10000), leaving little headroom.

Application‑Layer Check

Inspect the Jedis pool configuration (example shown):

<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
    <property name="maxTotal" value="200"/>   <!-- max connections -->
    <property name="maxIdle" value="50"/>    <!-- max idle -->
    <property name="minIdle" value="10"/>    <!-- min idle -->
    <property name="maxWaitMillis" value="3000"/>
    <property name="testOnBorrow" value="true"/>
</bean>

3000 application instances (15 nodes × maxTotal=200) lead to far more actual connections than expected.

Root‑Cause Analysis

Direct Cause: Connection Surge After Deployment

New version deployment set minIdle=10 on each node, causing each node to pre‑create 10 connections. Rolling update kept old and new nodes alive simultaneously, doubling the connection count. Some old nodes did not release connections gracefully, creating zombies.

Amplifying Factor: timeout=0

With timeout=0, idle connections are never closed, so zombie connections accumulate and occupy slots until the peak traffic arrives.

Snowball Chain

Redis connections full (maxclients=10000)
 → new requests cannot get a connection → JedisConnectionException
 → no circuit‑breaker → threads retry and block
 → web thread pool exhausted → health‑check timeout
 → node removed from registry → traffic shifts
 → remaining nodes’ Redis connections surge → also full
 → service becomes unavailable

Rapid Mitigation (Quick‑Bleed Solutions)

Increase maxclients Temporarily

# Temporary change (lost after restart)
$ redis-cli CONFIG SET maxclients 20000

Also raise system fd limits:

# Increase Redis process fd limit
$ prlimit --pid $(pidof redis-server) --nofile=30000
# Or adjust system limit
$ ulimit -n 65535

Enable timeout to Clean Zombie Connections

# Set idle timeout to 60 s (effective immediately)
$ redis-cli CONFIG SET timeout 60

After 60 s, idle connections > 60 s are closed automatically.

Bulk Kill Abnormal Connections

# Kill all normal client connections
$ redis-cli CLIENT KILL TYPE normal
# Kill connections from a specific IP range
$ redis-cli CLIENT KILL addr 10.0.1.12:0
# Ensure the command does not kill its own connection
$ redis-cli CLIENT KILL addr 10.0.1.12:0 skipme no

Restart Applications (as a fallback)

# Restart the application service (prefer rolling restart)
$ systemctl restart app-service

After restart, connection counts return to normal as minIdle connections are rebuilt.

Long‑Term Governance

Connection‑Pool Best Practices

<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
    <property name="maxTotal" value="50"/>      <!-- per node -->
    <property name="maxIdle" value="20"/>
    <property name="minIdle" value="5"/>
    <property name="maxWaitMillis" value="2000"/> <!-- 2 s -->
    <property name="testWhileIdle" value="true"/>
    <property name="timeBetweenEvictionRunsMillis" value="60000"/>
    <property name="blockWhenExhausted" value="true"/>
</bean>

maxTotal should be 20‑100 per node, based on actual concurrency.

maxWaitMillis 1000‑3000 ms prevents indefinite thread blocking.

Enable testWhileIdle to periodically verify idle connections.

Always return connections to the pool (try‑with‑resources or finally block).

Use a Connection Proxy (Twemproxy / Predixy)

When many clients connect directly to Redis, introduce a proxy layer to reduce total connections from M×N to M+N and provide buffering, read/write splitting, and failover.

Connection Limits and Firewall Protection

# Limit connections per application server (iptables)
$ iptables -A INPUT -p tcp --dport 6379 -m connlimit --connlimit-above 50 -j REJECT

Circuit‑Breaker and Retry Back‑off

// Resilience4j CircuitBreaker example
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .slidingWindowSize(10)
    .build();

// Simple exponential back‑off
int baseDelay = 100; // ms
for (int i = 0; i < maxRetries; i++) {
    try {
        return jedisPool.getResource();
    } catch (Exception e) {
        Thread.sleep(baseDelay * (long)Math.pow(2, i));
    }
}

Monitoring and Alerting

Key Redis metrics (exposed by redis_exporter) and recommended thresholds: redis_connected_clients / redis_config_maxclients > 80% → Warning redis_connected_clients / redis_config_maxclients > 90% → Critical redis_rejected_connections_total > 0 → P0 Emergency

Production Notes

CONFIG SET maxclients must be coordinated with ulimit -n, kernel fs.file-max, and /etc/security/limits.conf settings; missing any makes the change ineffective.

Do not modify timeout during peak traffic; a sudden drop to a small value can abruptly close many connections, causing massive reconnection storms.

When using CLIENT KILL, be aware of the skipme flag to avoid killing the command’s own connection.

Cloud Redis services may bind maxclients to instance size (e.g., 2 GB → 10000); verify limits before requesting increases.

If connection saturation persists without obvious zombie connections, check for connection leaks such as storing Jedis instances as member variables or missing close() in exception paths.

Conclusion

The outage was caused by three stacked factors: a deployment‑induced connection surge, timeout=0 allowing zombie connections to accumulate, and the lack of circuit‑breaker protection. Mitigation can be organized into three layers:

First layer – rapid bleed: increase maxclients, set a reasonable timeout, and optionally kill abnormal connections.

Second layer – root‑cause investigation: examine connection distribution, idle times, pool settings, and system limits.

Third layer – architectural governance: standardize pool configuration, implement circuit‑breakers, adopt connection proxies, set up monitoring/alerts, and define emergency runbooks.

Key lessons: never leave timeout=0 in production, size connection pools according to real concurrency, protect services with circuit‑breakers, and monitor connection metrics closely during deployments.