Why Did Our Redis Calls Take 1.2 Seconds? A Deep Dive into JedisPool Tuning

Background

On 2023‑03‑08 a load test in a certain region showed frequent request timeouts: P50 >400 ms, P90 >1.2 s, P99 >2 s. Detailed investigation revealed that Redis access latency spiked to about 1.2 seconds.

Possible Causes

Possibility 1: Server‑side issue? The Redis instance used was a high‑end redis_amber_master_4xlarge_multithread (16 CPU, 32 GB RAM, 480 GB SSD) with a theoretical QPS of 240 k and max connections 30 k, yet during peak load both QPS and load remained low.

Possibility 2: Physical network issue? Bandwidth measurements showed the request traffic was far below the NIC capacity, and retransmission rates were normal.

Possibility 3: Client‑side issue? The investigation therefore focused on the client.

Client‑side Findings

JVM Full GC? Monitoring (ARMS) showed increased YGC count and duration but no Full GC events.

JedisPool problems

maxBorrowWaitTimeMills was too large – the thread waited up to 1200 ms to obtain a connection from the pool, causing request slowdown.

Excessive connection creation/destruction: createdCount = 11555, destroyedCount = 11553, indicating an unreasonable max-idle setting that caused idle objects to be destroyed and recreated on each return.

These metrics are maintained by the JedisPool object for the lifetime of the JVM, so a post‑mortem heap dump can reveal them without reproducing the load.

Configuration Details

spring.redis.jedis.pool.max-active=100
spring.redis.jedis.pool.max-idle=16
spring.redis.jedis.pool.time-between-eviction-runs-millis=30000
spring.redis.jedis.pool.min-idle=0
spring.redis.jedis.pool.test-while-idle=true
spring.redis.jedis.pool.num-tests-per-eviction-run=-1
spring.redis.jedis.pool.min-evictable-idle-time-millis=60000

Explanation:

max‑active (mapped to ObjectPool’s maxTotal) limits total connections to 100.

max‑idle of 16 means idle objects ≥ 16 are destroyed on return.

A background thread runs every 30 seconds to perform keep‑alive and eviction checks.

min‑idle of 0 means the pool does not maintain a core size.

Pulsed Request Problem

During a short burst (T2‑T3) 84 connections were created and immediately destroyed, incurring huge overhead.

Desired Behavior

Set max‑active, max‑idle, and min‑idle to the same stable value to avoid on‑the‑fly connection creation.

Enable periodic keep‑alive and eviction to clean up stale connections.

Avoid rebuilding connections merely because they have been idle too long.

spring.redis.jedis.pool.max-active=500   // 4 nodes × 500 = 2000 < Redis capacity
spring.redis.jedis.pool.max-idle=500
spring.redis.jedis.pool.time-between-eviction-runs-millis=30000
spring.redis.jedis.pool.min-idle=500
spring.redis.jedis.pool.test-while-idle=true
spring.redis.jedis.pool.num-tests-per-eviction-run=-1   // test all 500 each run
spring.redis.jedis.pool.min-evictable-idle-time-millis=-1 // prevent mass reconnection

Result Verification

A subsequent load test on 2023‑04‑13 with the same traffic model showed:

maxBorrowWaitTimeMills reduced by ~80 %.

createdCount dropped from 11 555 to 500 (the pool’s initial size).

Overall service latency improved dramatically: P50 and P90 fell by ~60 %, P99 by ~70 %.