Why Was My Redis Access Latency So High? A Deep Dive into JedisPool Tuning

Background

During an online stress test on 2023‑03‑08, the service experienced frequent request timeouts with severe performance degradation: P50 > 400 ms, P90 > 1.2 s, P99 > 2 s. Detailed analysis showed Redis access latency spiking to around 1.2 seconds.

Why Was Redis Access Latency So High?

The simplified Redis access flow is illustrated below:

Possibility 1: Server‑side Issue?

Redis instance: redis_amber_master_4xlarge_multithread (16 C/32 G + 480 GB SSD), max QPS ~240 k, max connections 30 k – a very powerful configuration.

During peak load, both QPS and overall load remained low.

Possibility 2: Physical Network Issue?

Network bandwidth was far from saturated, and metrics such as NIC retransmission rates were normal.

Possibility 3: Client‑side Issue?

The problem was traced to the client, specifically the JedisPool configuration.

JVM Full GC (FGC) STW?

Monitoring showed an increase in YGC count and duration, but no full GC occurred.

JedisPool Issues

maxBorrowWaitTimeMills too large : the client waited up to 1200 ms to obtain a connection from the pool, causing request slowdown.

Excessive connection creation/destruction : createdCount = 11555, destroyedCount = 11553. An unreasonable max-idle setting caused idle connections to be destroyed, leading to frequent TCP connection churn.

These metrics are maintained by the JedisPool object for the lifetime of the JVM, so they persist across tests.

JedisPool Parameter Overview

spring.redis.jedis.pool.max-active=100</code><code>spring.redis.jedis.pool.max-idle=16</code><code>spring.redis.jedis.pool.time-between-eviction-runs-millis=30000</code><code>spring.redis.jedis.pool.min-idle=0</code><code>spring.redis.jedis.pool.test-while-idle=true</code><code>spring.redis.jedis.pool.num-tests-per-eviction-run=-1</code><code>spring.redis.jedis.pool.min-evictable-idle-time-millis=60000

Key interpretations:

max-active : total pool size (idle + active) maps to ObjectPool.maxTotal.

max-idle : if idle objects exceed 16, they are destroyed on return.

Background thread runs every 30 s for heartbeat and health checks.

min-idle : set to 0, so the pool does not maintain a permanent core size.

Problems Caused by Burst Requests

During a burst (T2‑T3), 84 connections were created and immediately destroyed, incurring high overhead.

Desired Behavior

Set pool max and min to a stable value to avoid on‑the‑fly connection creation.

Enable periodic heartbeat to promptly remove stale or timed‑out connections.

Avoid destroying idle connections solely due to idle time, preventing unnecessary large‑scale reconnections.

spring.redis.jedis.pool.max-active=500 // 4 nodes × 500 = 2000, far below Redis capacity</code><code>spring.redis.jedis.pool.max-idle=500</code><code>spring.redis.jedis.pool.time-between-eviction-runs-millis=30000 // heartbeat</code><code>spring.redis.jedis.pool.min-idle=500 // stable pool size</code><code>spring.redis.jedis.pool.test-while-idle=true</code><code>spring.redis.jedis.pool.num-tests-per-eviction-run=-1 // test all 500 connections each run</code><code>spring.redis.jedis.pool.min-evictable-idle-time-millis=-1 // prevent idle‑time based eviction

Effect Verification

After applying the new configuration and re‑running the stress test on 2023‑04‑13, the results were:

maxBorrowWaitTimeMills reduced by ~80%.

createdCount dropped from 11 555 to 500 (initial pool size).

Overall business performance improved dramatically: P50 and P90 decreased by ~60%, P99 by ~70%.