How We Diagnosed and Fixed a Redisson Bug in Production

Background

Java project uses Redisson (sentinel mode) to connect to Redis. A brief AWS network jitter (~10 min) caused some Redis requests to remain slow after network recovery, blocking Tomcat IO threads.

Investigation Stages

Stage 1 – Blind retry

Initial attempts to reproduce failed due to lack of observability, limited knowledge of Redisson internals, and the issue affecting only a subset of requests.

Stage 2 – Source code review

Identified four potentially slow points in Redisson’s request‑response flow:

Acquiring a connection when the pool is exhausted or at its limit (

org.redisson.connection.pool.ConnectionPool#acquireConnection

).

Sending a Redis command and waiting for Netty writeFuture ( org.redisson.command.CommandAsyncService#sendCommand and org.redisson.command.CommandAsyncService#checkWriteFuture).

Waiting for Netty event loop to deliver the response ( org.redisson.client.handler.CommandDecoder#completeResponse).

Triggering retries via a timeout handler when any of the above fails ( org.redisson.command.CommandAsyncService#async, attempt increment).

Stage 3 – Metric instrumentation

A Java‑agent built with ASM instrumented the four locations. Prometheus‑exposed metrics visualized in Grafana: retry – current retry count for a command. timeout – latency from successful send to response. waitQueue – number of commands waiting for a connection when the pool limit is reached.

Connection‑pool statistics: freeConnections, allConnections, isFreezed, etc.

Stage 4 – Chaos engineering & load testing

Using ChaosBlade, 50 % packet loss and 200 ms latency were injected on the Redis port. Grafana showed that after the jitter retry kept increasing, waitQueue returned to zero, and timeout remained flat.

Arthas monitor (with a PR adding condition‑express support, merged in Arthas 3.4.0) and stack commands traced async calls with attempt > 0. Calls originated from org.redisson.command.CommandAsyncService$6.run(). Further inspection revealed that the timeout occurred after Netty confirmed command transmission, not during connection acquisition.

Stage 5 – Deep dive & communication with maintainers

Redisson maintainers pointed to ConnectionWatchdog + PingConnectionHandler for reconnection, which did not explain the observed behavior.

Code review uncovered a closed flag in RedisConnection that is set only by closeAsync() and is independent of the Netty channel state. The pool’s add() logic does not check this flag, allowing closed connections to be re‑added to freeConnections. A heap dump confirmed many connections with closed=true remained in the pool.

Stage 6 – Fix

The fix adds a check for RedisConnection.closed in releaseConnection() before returning a connection to freeConnections. The change was verified locally, submitted as PR #2956, and merged into Redisson master for the 3.13.3 milestone.

Issue: https://github.com/redisson/redisson/issues/2947

PR: https://github.com/redisson/redisson/pull/2956