Troubleshooting High Redis Client Connections: A Step-by-Step Guide to Identification and Optimization

1. Problem Background

In production environments, Redis serves as a core component for caching and session storage, and its stability directly impacts business availability. Monitoring alerts showed Redis client connections continuously rising, even hitting the maxclients limit, causing new connections to be rejected and business timeouts and errors.

High Redis connection counts can stem from multiple causes: connections not properly released, misconfigured connection pools, slow queries blocking connections, sudden traffic spikes, connection leaks, or attack traffic. Quickly identifying the root cause, taking targeted measures to restore service, and improving monitoring and prevention mechanisms are core Redis operational skills.

This article provides a practical, systematic troubleshooting methodology for high Redis connection counts, covering investigation steps, localization methods, common causes, optimization techniques, and production best practices to help operations engineers build complete problem-handling capabilities.

2. Applicable Scenarios

Abnormal growth in Redis client connections

Redis rejecting new connections with max number of clients reached errors

Application timeouts or failures connecting to Redis

Persistently high Redis connection counts

Need to optimize Redis connection pool configuration

Need to troubleshoot Redis connection leaks

Need to handle connection surges from burst traffic

Applicable to standalone Redis, master-replica, and Redis Cluster architectures.

3. Core Concepts

3.1 Redis Connection Model

Redis uses a single-threaded model to process client requests but can maintain multiple client connections simultaneously. Each connection consumes memory and file descriptors.

Connection Establishment Flow

Client initiates connection → TCP three-way handshake → Redis accepts connection → Client sends command → Redis processes and returns → Client receives response

Connection Types

Regular client connections: Application-established connections

Replica connections: Replicas connecting to master during replication

Sentinel connections: Monitoring connections in Sentinel mode

Pub/Sub connections: Connections for publish/subscribe mode

3.2 Connection-Related Configuration

maxclients

Maximum client connections, default 10000. Redis rejects new connections when limit reached.

# View current config
redis-cli CONFIG GET maxclients

# Temporary change (lost on restart)
redis-cli CONFIG SET maxclients 20000

# Permanent change (edit redis.conf)
maxclients 20000

timeout

Client idle timeout in seconds; idle connections exceeding this are closed. Default 0 means no timeout.

# View current config
redis-cli CONFIG GET timeout

# Set 300-second timeout
redis-cli CONFIG SET timeout 300

tcp-keepalive

TCP keepalive interval in seconds to detect dead connections. Default 300.

# View current config
redis-cli CONFIG GET tcp-keepalive

# Set 60 seconds
redis-cli CONFIG SET tcp-keepalive 60

3.3 Connection Statistics

# View current connections
redis-cli INFO clients | grep connected_clients

# View historical max connections
redis-cli INFO stats | grep maxclients

# View rejected connections
redis-cli INFO stats | grep rejected_connections

Example output:

connected_clients:1523
rejected_connections:0

3.4 Common Cause Categories

Application layer: Unreleased connections, misconfigured pools, connection leaks

Redis layer: Slow queries blocking, large key operations, persistence blocking

Network layer: Network jitter causing connection pileup, firewall restrictions

Business layer: Sudden concurrency spikes, scheduled tasks running simultaneously

Attacks: Malicious connections, DDoS

4. Complete Troubleshooting Process

Phase 1: Confirm the Problem

Check Current Connection Count

redis-cli INFO clients

Output:

# Clients
connected_clients:9856
client_recent_max_input_buffer:8
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

Key metrics: connected_clients: Current connection count blocked_clients: Clients blocked on BLPOP, BRPOP, etc. clients_in_timeout_table: Clients waiting for timeout

Check maxclients Configuration

redis-cli CONFIG GET maxclients

Output:

1) "maxclients"
2) "10000"

If connected_clients approaches maxclients, connections are near the limit.

Check Rejected Connections

redis-cli INFO stats | grep rejected_connections

Output: rejected_connections:145 If non-zero and growing, connections are being rejected.

Initial Assessment

Current connections: 9856, limit: 10000, at 98.5%

145 connections rejected

Problem confirmed, immediate action needed

Next step: Identify connection sources and specific clients.

Phase 2: Examine Client List

Use CLIENT LIST

redis-cli CLIENT LIST

Sample output (partial):

id=12345 addr=192.168.1.10:54321 fd=8 name= age=3600 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=12346 addr=192.168.1.10:54322 fd=9 name= age=3590 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=12347 addr=192.168.1.11:43210 fd=10 name= age=120 idle=120 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL

Key fields: id: Unique client identifier addr: Client IP and port age: Connection lifetime (seconds) idle: Idle time (seconds) flags: Client flags (N=normal, S=replica, O=monitor, etc.) cmd: Last executed command

Count Connections per IP

redis-cli CLIENT LIST | awk '{print $2}' | awk -F'=' '{print $2}' | awk -F':' '{print $1}' | sort | uniq -c | sort -rn

Output:

5230 192.168.1.10
3156 192.168.1.11
987 192.168.1.12
483 192.168.1.13

Analysis

192.168.1.10 holds 5230 connections, 53% of total

Need to identify the application behind this IP and root cause

Count Long-Idle Connections

redis-cli CLIENT LIST | awk '$12 ~ /idle=[0-9]+/ {match($12, /idle=([0-9]+)/, arr); if (arr[1] > 300) print $0}' | wc -l

Output: 2345 connections idle over 300 seconds (5 minutes), likely zombie connections.

Next step: Identify the application at 192.168.1.10 and investigate its high connection usage.

Phase 3: Locate the Application

Map IP to Server

Assume 192.168.1.10 is application server app-server-01.

SSH to Application Server

ssh app-server-01

View Connections to Redis

netstat -antp | grep 6379

Output (partial):

tcp     0     0 192.168.1.10:54321     192.168.2.100:6379     ESTABLISHED 12345/java
...

All connections originate from the same Java process (PID 12345).

Count Connections

netstat -antp | grep 6379 | wc -l

Output: 5230 — confirming this process holds 5230 Redis connections.

Verify Process Details

ps -fp 12345

Output:

UID        PID  PPID  C STIME TTY          TIME CMD
app      12345     1  25 Feb14 ?        00:123:45 java -jar /opt/app/myapp.jar

Confirmed as application myapp.jar.

Next step: Analyze application code and configuration for connection pool settings and usage patterns.

Phase 4: Analyze Application Connection Pool Configuration

Check Application Config

Assuming Jedis pool, application.yml:

spring:
  redis:
    host: 192.168.2.100
    port: 6379
    jedis:
      pool:
        max-active: 8
        max-idle: 8
        min-idle: 2
        max-wait: 3000
        timeout: 3000

Analysis

max-active: 8

— maximum active connections set to 8

Actual connections: 5230, far exceeding configuration

Possible Causes

Multiple application instances or threads each creating their own pool

Connections not properly returned, causing pool to create new ones

Connection leaks in application code

Check Application Logs

tail -f /var/log/app/app.log | grep -i redis

Output:

2024-02-15 10:23:45 ERROR Could not get a resource from the pool
2024-02-15 10:23:46 WARN Connection pool exhausted, creating new connection
2024-02-15 10:23:47 ERROR Could not get a resource from the pool

Key findings: Pool exhausted, new connections created continuously.

Next step: Examine application code for connection usage correctness.

Phase 5: Inspect Application Code

Review Redis Call Code

Incorrect example:

// Incorrect example
public void incorrectUsage() {
    Jedis jedis = new Jedis("192.168.2.100", 6379);
    String value = jedis.get("key");
    // Forgot to close connection
    // jedis.close();
}

Problem Analysis

New Jedis object created on every call

Connection not closed after use

Causes connection leak, unbounded connection growth

Correct Example — Using Connection Pool

// Correct example
public void correctUsage() {
    try (Jedis jedis = jedisPool.getResource()) {
        String value = jedis.get("key");
    } // try-with-resources auto-closes
}

Or manual close:

public void correctUsageManual() {
    Jedis jedis = null;
    try {
        jedis = jedisPool.getResource();
        String value = jedis.get("key");
    } finally {
        if (jedis != null) {
            jedis.close();
        }
    }
}

Root Cause Confirmed

Application code leaks connections by not using a pool or not closing connections properly.

Next step: Fix code and deploy, while applying temporary mitigations.

Phase 6: Temporary Mitigation Measures

Option 1: Clean Idle Connections

Find connections idle >5 minutes:

redis-cli CLIENT LIST | awk '$12 ~ /idle=[0-9]+/ {match($12, /idle=([0-9]+)/, arr); if (arr[1] > 300) print $2}' | awk -F'=' '{print $2}'

Output:

192.168.1.10:54500
192.168.1.10:54501
...

Kill them:

redis-cli CLIENT LIST | awk '$12 ~ /idle=[0-9]+/ {match($0, /id=([0-9]+)/, id); match($0, /idle=([0-9]+)/, idle); if (idle[1] > 300) print id[1]}' | xargs -I {} redis-cli CLIENT KILL ID {}

Risks: May affect legitimate long-lived connections; confirm impact scope first.

Option 2: Set Idle Timeout

redis-cli CONFIG SET timeout 300

Redis will auto-close connections idle >300 seconds.

Risks: May affect idle-but-needed connections; ensure application has reconnection logic.

Option 3: Temporarily Raise maxclients

redis-cli CONFIG SET maxclients 20000

Only a stopgap; does not fix root cause.

Option 4: Restart Leaking Application

ssh app-server-01
sudo systemctl restart myapp

Connections drop to zero, but will recur if code unchanged.

Results

After cleaning idle connections, count dropped from 9856 to ~7500. Setting timeout 300 enabled automatic cleanup.

Next step: Fix application code and deploy.

Phase 7: Code Fix

Fix Strategy

Use connection pool and ensure proper closure.

Fixed Code

// Configure connection pool
@Configuration
public class RedisConfig {
    @Bean
    public JedisPool jedisPool() {
        JedisPoolConfig config = new JedisPoolConfig();
        config.setMaxTotal(100);
        config.setMaxIdle(50);
        config.setMinIdle(10);
        config.setMaxWaitMillis(3000);
        config.setTestOnBorrow(true);
        return new JedisPool(config, "192.168.2.100", 6379, 3000);
    }
}

// Use connection pool
@Service
public class CacheService {
    @Autowired
    private JedisPool jedisPool;

    public String get(String key) {
        try (Jedis jedis = jedisPool.getResource()) {
            return jedis.get(key);
        }
    }

    public void set(String key, String value) {
        try (Jedis jedis = jedisPool.getResource()) {
            jedis.set(key, value);
        }
    }
}

Unit Test

@Test
public void testConnectionPool() {
    CacheService service = new CacheService();
    ExecutorService executor = Executors.newFixedThreadPool(100);
    for (int i = 0; i < 1000; i++) {
        executor.submit(() -> {
            service.set("key" + i, "value" + i);
            String value = service.get("key" + i);
            assertEquals("value" + i, value);
        });
    }
    executor.shutdown();
    executor.awaitTermination(1, TimeUnit.MINUTES);
    JedisPoolConfig config = jedisPool.getPoolConfig();
    assertTrue(jedisPool.getNumActive() <= config.getMaxTotal());
}

Deployment Process

Commit code and code review

Canary deploy to test environment

Monitor connection count changes

Canary to 20% production traffic

Observe connections, confirm fix

Full rollout

Timeline

Feb 15 14:00 — Code fix merged

Feb 15 16:00 — Canary to production

Feb 15 18:00 — Full rollout

Phase 8: Verify Effectiveness

Check Connection Count

redis-cli INFO clients | grep connected_clients

Output: connected_clients:156 — down from 9856, normal.

Check Rejected Connections

redis-cli INFO stats | grep rejected_connections

Output: rejected_connections:145 — no further growth, no new rejections.

Check Application Server Connections

ssh app-server-01
netstat -antp | grep 6379 | wc -l

Output: 15 — single instance maintains 15 connections, matching pool config.

Ongoing Observation

24-hour monitoring shows stable connection counts; issue resolved.

Phase 9: Postmortem

Incident Timeline

Time    Event
10:00   Redis connection count alert received
10:05   Confirmed connections near limit
10:10   Traced majority to 192.168.1.10
10:15   Identified application myapp.jar
10:20   Code analysis revealed connection leak
10:30   Cleaned idle connections, set timeout
10:45   Connections dropped to safe range
14:00   Code fix merged
16:00   Canary deployment
18:00   Full deployment

Root Cause

Application code did not use a connection pool; each call created a new connection and failed to close it, causing a leak.

Preventive Measures

Short-term

Fix application code to use connection pool

Set Redis idle connection timeout

Add connection count monitoring and alerts

Long-term

Establish code review process checking resource management

Introduce static analysis tools to detect resource leaks

Improve unit tests covering concurrency and resource handling

Define Redis connection pool configuration standards

Enhance monitoring: connection count, sources, idle connections

Schedule regular Redis connection health checks

Lessons Learned

All external resources (DB, Redis, HTTP) must be properly closed

Connection pools are best practice; avoid frequent create/destroy

Monitor not just total connections but also sources and idle counts

Temporary mitigations restore service quickly but cannot replace root-cause fixes

Code reviews and static analysis catch resource leaks early

5. Common Causes and Investigation Methods

Cause 1: Connections Not Released

Symptoms

Connections grow continuously

Many idle connections

App logs show pool exhaustion

Investigation

# View idle connections
redis-cli CLIENT LIST | awk '$12 ~ /idle=[0-9]+/ {match($12, /idle=([0-9]+)/, arr); if (arr[1] > 60) print $0}' | wc -l

Fix

Use try-with-resources or finally to ensure closure

Use connection pool and properly acquire/release connections

Cause 2: Misconfigured Connection Pool

Symptoms

Large connection count fluctuations

Pool exhaustion at peak

High connections even at low traffic

Investigation

Review pool config and actual usage.

Fix

Adjust pool parameters:

spring:
  redis:
    jedis:
      pool:
        max-active: 100  # max active connections
        max-idle: 50     # max idle connections
        min-idle: 10     # min idle connections
        max-wait: 3000   # acquire timeout (ms)

Rule of thumb: max-active: concurrency + 20% max-idle: 50%-80% of max-active min-idle: typical concurrent load max-wait: 3-5 seconds

Cause 3: Slow Queries Blocking

Symptoms

Sudden connection spike

Many connections waiting

High Redis CPU

Investigation

# View slow queries
redis-cli SLOWLOG GET 10

Output example:

1) 1) (integer) 100
   2) (integer) 1708012345
   3) (integer) 5234567
   4) 1) "KEYS"
      2) "user:*"

Key: Query #100 took 5.23 seconds, command KEYS user:*.

Fix

Optimize slow queries; avoid O(N) commands like KEYS

Use SCAN instead of KEYS

Avoid large key operations

Set timeouts for slow queries

Cause 4: Concurrency Surge

Symptoms

Sudden connection increase

Short duration

Correlates with business peaks or events

Investigation

Check business metrics for traffic spike.

Fix

Scale Redis cluster

Optimize architecture: rate limiting, degradation

Capacity planning and stress testing beforehand

Cause 5: Frequent Application Restarts

Symptoms

Periodic connection count fluctuations

App logs show frequent restarts

Investigation

Examine restart logs and causes.

Fix

Fix underlying restart causes

Optimize startup to reduce time

Use connection pre-warming to avoid burst creation at startup

6. Monitoring and Alerting

6.1 Connection Monitoring

Using Redis INFO

redis-cli INFO clients | grep connected_clients

Using Prometheus + Redis Exporter

Exported metrics:

redis_connected_clients
redis_rejected_connections_total
redis_blocked_clients

Grafana panels:

Current connections

Connection trend

Rejected connections

Blocked clients

6.2 Alert Rules

Connections Near Limit

alert: RedisConnectionsHigh
expr: redis_connected_clients / redis_config_maxclients > 0.8
for: 5m
annotations:
  summary: "Redis Connections High"
  description: "Redis {{ $labels.instance }} connections {{ $value | humanizePercentage }} exceed threshold"

Rapid Connection Growth

alert: RedisConnectionsIncreasing
expr: rate(redis_connected_clients[5m]) > 10
for: 5m
annotations:
  summary: "Redis Connections Increasing Rapidly"
  description: "Redis {{ $labels.instance }} connections growing at {{ $value }} per second"

Connections Rejected

alert: RedisConnectionsRejected
expr: increase(redis_rejected_connections_total[5m]) > 0
for: 1m
annotations:
  summary: "Redis Rejecting Connections"
  description: "Redis {{ $labels.instance }} rejected {{ $value }} connections"

6.3 Connection Source Monitoring

Periodic script to tally sources:

#!/bin/bash
REDIS_HOST="127.0.0.1"
REDIS_PORT="6379"
redis-cli -h $REDIS_HOST -p $REDIS_PORT CLIENT LIST | \
  awk '{print $2}' | \
  awk -F'=' '{print $2}' | \
  awk -F':' '{print $1}' | \
  sort | uniq -c | sort -rn | \
  head -10

Output to monitoring system or logs.

7. Production Best Practices

7.1 Connection Pool Configuration

Use pools; avoid frequent create/destroy

Size pools based on concurrency

Set reasonable timeouts

Enable validity checks (testOnBorrow)

7.2 Resource Management

Use try-with-resources or finally for closure

Avoid creating connections in loops

Avoid holding connections long-term

Release unused connections promptly

7.3 Redis Configuration

Set maxclients per actual needs

Configure reasonable idle timeout

Enable TCP keepalive

Monitor connections and rejections

7.4 Application Architecture

Implement rate limiting, degradation, circuit breaking

Avoid synchronous waits; consider async

Capacity planning and stress testing

Prepare for traffic bursts

7.5 Monitoring and Alerting

Monitor connections, sources, idle counts

Set sensible alert thresholds

Regular connection health inspections

Retain historical data for analysis

8. Common Command Reference

8.1 View Connection Info

# Current connections
redis-cli INFO clients | grep connected_clients

# All clients
redis-cli CLIENT LIST

# Client details
redis-cli CLIENT INFO

# Connections per IP
redis-cli CLIENT LIST | awk '{print $2}' | awk -F'=' '{print $2}' | awk -F':' '{print $1}' | sort | uniq -c | sort -rn

# Idle connections >300s
redis-cli CLIENT LIST | awk '$12 ~ /idle=[0-9]+/ {match($12, /idle=([0-9]+)/, arr); if (arr[1] > 300) print $0}'

8.2 Manage Connections

# Kill by IP:port
redis-cli CLIENT KILL 192.168.1.10:54321

# Kill by ID
redis-cli CLIENT KILL ID 12345

# Kill all normal clients
redis-cli CLIENT KILL TYPE normal

# Set client name
redis-cli CLIENT SETNAME myclient

# Get client name
redis-cli CLIENT GETNAME

8.3 Configuration Management

# View maxclients
redis-cli CONFIG GET maxclients

# Set maxclients
redis-cli CONFIG SET maxclients 20000

# View timeout
redis-cli CONFIG GET timeout

# Set timeout
redis-cli CONFIG SET timeout 300

# View tcp-keepalive
redis-cli CONFIG GET tcp-keepalive

# Set tcp-keepalive
redis-cli CONFIG SET tcp-keepalive 60

8.4 Slow Queries

# View slow log
redis-cli SLOWLOG GET 10

# Slow log length
redis-cli SLOWLOG LEN

# Reset slow log
redis-cli SLOWLOG RESET

9. Summary

High Redis client connections are a common production issue. Troubleshooting and optimization require:

Core Troubleshooting Path

Confirm anomaly → View client list → Count sources → Locate app → Analyze pool config → Check code → Apply temporary mitigations → Fix root cause → Verify → Improve monitoring and prevention

Key Takeaways

Use CLIENT LIST to pinpoint sources

Count per-IP and idle connections

Inspect app pool config and code

Temporary relief: clean idle, set timeout, raise limit

Permanent fix: code repair, config optimization, architecture improvement

Establish monitoring, alerting, and inspection routines

Best Practices

Application: Proper pool usage, ensure closure

Redis: Sensible maxclients and timeout

Monitoring: Track connections, sources, rejections

Architecture: Rate limiting, degradation, async, capacity planning

Redis connection management is a foundational operational skill requiring holistic consideration across application, Redis, monitoring, and architecture layers to guarantee production stability.