Why Redis Failed: Jedis Misconfigurations That Spark Service Avalanches

Background

Redis is the de‑facto remote‑cache solution for many Internet services, and Jedis is one of the most widely used Java clients. The author’s project runs Redis 3.x in cluster mode (multiple nodes with master‑slave pairs) and accesses it through Jedis.

A physical‑machine failure caused a node in the Redis cluster to perform a master‑slave switch. During the switch Jedis’ retry mechanism was triggered, eventually leading to a service‑wide avalanche.

Fault Record

Message‑queue backlog alert (timestamp 2022‑11‑29 23:50:21, queue size 159 412, threshold > 100 000).

System monitoring showed a sharp drop in request volume and average interface latency approaching 60 seconds.

Thread‑wait count surged dramatically during the incident.

Operations confirmed that a Redis master‑slave switch coincided with the outage.

Failure Analysis

Traffic Drop

NGINX logs contained many “connection timed out” errors, which caused NGINX to mark the backend as unavailable and emit “no live upstreams”. This prevented request forwarding and resulted in a steep traffic decline.

Latency Issue

Jedis threw connect timed out exceptions while acquiring connections. The default connection timeout is DEFAULT_TIMEOUT = 2000 ms. Each retry adds roughly 2 seconds; with six retries a single request can take about 12 seconds.

Because the business logic performs five Redis calls per request, the total latency can reach ~60 seconds, matching the observed average.

Jedis Execution Flow

The following diagram shows the high‑level flow of a Jedis command:

Key steps:

Obtain a connection based on the slot calculated by JedisClusterCRC16.getSlot.

Execute the command via execute.

If connection acquisition or command execution fails, an exception triggers the retry logic.

Source Code Highlights

public class JedisCluster extends BinaryJedisCluster implements JedisCommands, MultiKeyJedisClusterCommands, JedisClusterScriptingCommands {
    @Override
    public String set(final String key, final String value, final String nxxx, final String expx, final long time) {
        return new JedisClusterCommand<String>(connectionHandler, maxAttempts) {
            @Override
            public String execute(Jedis connection) {
                return connection.set(key, value, nxxx, expx, time);
            }
        }.run(key);
    }
}

The runWithRetries method encapsulates the retry logic, repeatedly calling connectionHandler.getConnectionFromSlot until the maximum attempt count is reached.

private T runWithRetries(byte[] key, int attempts, boolean tryRandomNode, boolean asking) {
    Jedis connection = null;
    try {
        // obtain connection
        connection = connectionHandler.getConnectionFromSlot(JedisClusterCRC16.getSlot(key));
        return execute(connection);
    } catch (JedisConnectionException e) {
        if (attempts <= 1) {
            connectionHandler.renewSlotCache();
            throw e;
        }
        return runWithRetries(key, attempts - 1, tryRandomNode, asking);
    } finally {
        releaseConnection(connection);
    }
}

Retry Mechanism

If an exception occurs while obtaining a connection or executing a command, Jedis retries up to maxAttempts times. In the incident the client performed six retries, each lasting about 2 seconds, leading to the observed 12‑second per‑request delay.

Recommendations

Configure Jedis parameters according to the actual workload:

maxAttempts : maximum number of retries.

connectionTimeout : connection timeout (e.g., 100 ms in production).

soTimeout : socket read timeout.

In the author’s production environment the settings are connectionTimeout = 100 ms, soTimeout = 100 ms, and maxAttempts = 2. With these values, even if a Redis node fails, a request typically completes within 1 second, preventing a cascade failure.

Conclusion

The analysis links a Redis master‑slave switch to a service avalanche caused by overly aggressive Jedis timeout and retry settings. Properly tuning connectionTimeout, soTimeout, and maxAttempts can dramatically reduce latency during node failures and keep the service stable.