Why Did Our API Hang? Uncovering Redis Connection Pool Blocking in Spring Boot

In a private sandbox environment an API became unresponsive for a whole week, with all requests hanging and no error logs from the database or Redis. Initial attempts to restart the application temporarily restored service, but the problem recurred more frequently.

Initial Investigation

Running top -H -p 12798 on the affected server showed no obvious CPU spikes. A jstack of the process revealed several threads in a LOCK state, but none pointed to business logic.

Deep Dive with Arthas

After installing the Alibaba open‑source Java diagnostic tool Arthas, the thread command exposed many http-nio Tomcat threads stuck in waiting state. Further inspection of a specific thread’s stack trace showed the block occurring during a Redis command.

Analyzing the Redis Connection Code

The application used Spring’s JedisConnectionFactory. The relevant method is:

protected Jedis fetchJedisConnector() {
    try {
        if (usePool && pool != null) {
            return pool.getResource();
        }
        Jedis jedis = new Jedis(getShardInfo());
        jedis.connect();
        return jedis;
    } catch (Exception ex) {
        throw new RedisConnectionFailureException("Cannot get Jedis connection", ex);
    }
}

When pool.getResource() is called, the underlying GenericObjectPool executes borrowObject(), which eventually calls borrowObject(long borrowMaxWaitMillis). The code contains a loop that waits while borrowMaxWaitMillis < 0, causing an indefinite wait if maxWaitMillis is not configured.

Root Cause

The Redis pool configuration lacked a maxWaitMillis setting, so borrowMaxWaitMillis defaulted to -1. Consequently, threads attempting to obtain a Jedis connection entered an endless wait, blocking the Tomcat request threads and producing 500 errors without explicit exceptions.

Fix Implemented

Adding a proper timeout to the pool resolved the deadlock:

JedisPoolConfig config = new JedisPoolConfig();
config.setMaxWaitMillis(2000); // 2 seconds
JedisConnectionFactory factory = new JedisConnectionFactory();
factory.setPoolConfig(config);
factory.afterPropertiesSet();

After redeploying the service, the API behaved normally, and subsequent reproductions showed no hanging threads.

Best Practices

Avoid calling stringRedisTemplate.getConnectionFactory().getConnection() directly; use RedisCallback or let Spring manage the connection lifecycle.

Always release a Redis connection back to the pool, e.g., via RedisConnectionUtils.releaseConnection(conn, factory).

Configure pool parameters such as maxWaitMillis, maxTotal, and maxIdle to prevent resource starvation.

Prefer higher‑level RedisTemplate operations; resort to low‑level commands only when necessary (e.g., SCAN, SETNX).

Avoid using the KEYS command in production as it can cause performance issues.

By correcting the pool configuration and using proper connection handling, the API freeze was eliminated and the system became stable.