From Network Jitter to Fault Drills: A Real‑World Redisson Bug Investigation

Background

Rapid evolution of the business and technical architecture increased service availability challenges, raising fault costs and lowering user tolerance for outages. The team needed to maintain stable user experience, reduce tangible and intangible loss from major incidents, and improve overall emergency response capability.

Fault Introduction

During a sunny afternoon, users reported slow responses, followed by system crashes. Alerts triggered a rapid‑recovery phase: the client‑side Tomcat thread pool was exhausted, leading to a partial restart of client services, which temporarily restored service but failed again. Subsequent APM tracing showed high latency in Redis calls from lower‑layer services. Restarting the product service yielded the same transient effect. A rollback of a recent code version also only temporarily fixed the issue.

After internal investigation failed to resolve the problem, the team consulted the cloud platform, which reported network jitter during the incident. The jitter was suspected as the root cause.

Fault Review

Observed symptom: partial user slowdown, then full system crash.

Root cause trigger: network jitter on the cloud platform.

Open questions included why only part of the traffic was affected initially and why service restarts were only temporarily effective.

Investigation Process

1. Log Analysis

Log inspection revealed numerous Dubbo timeout errors (default 1000 ms). All timed‑out interfaces called Redis, prompting suspicion of Redis‑related latency.

2. APM Full‑Link Tracing

APM showed Redis call latency exceeding 1500 ms for some requests during the fault period.

3. Monitoring

Redis slow‑log check found no slow entries.

Resource metrics (CPU, memory, CPS) were within normal ranges.

Thus, Redis itself showed no abnormality.

4. Synthesis

Log analysis identified Dubbo timeouts linked to Redis calls.

APM confirmed abnormal Redis latency.

Monitoring ruled out Redis server issues.

The discrepancy suggested a problem in the middleware handling Redis interactions—specifically Redisson, the Java Redis client.

Test Design

To reproduce the issue, the team designed fault‑injection scenarios targeting the Redis cluster (Sentinel high‑availability mode) and its client connections.

Fault injection points: input ports of slave, master, and sentinel nodes (6379) with 50 % packet loss and 200 ms lag.

Expected outcomes included client request failures, sentinel‑driven node off‑line notifications, and potential master/slave switch complications.

Using JMeter for load generation and a chaos‑engineering platform (built on open‑source ChaosBlade) for fault injection, the experiment proceeded as follows:

Define a test API get_redis_cache_api that fetches data from Redis.

Run continuous JMeter requests.

Inject the defined network faults.

Results for the slave‑node injection:

During fault injection, some client requests timed out or errored.

After fault cessation, a mix of normal and timed‑out responses persisted.

Redis metrics remained normal and no slow logs appeared.

These observations matched the production incident.

Root‑Cause Confirmation

Further analysis compared the number of OS‑level Redis connections with the connections maintained by Redisson. A mismatch was found: Redisson’s connection pool contained closed connections that were still being used after the network jitter subsided.

The conclusion: network jitter caused temporary loss of connectivity between Sentinel and Redis nodes; Redisson’s handling of the reconnection left stale connections in its pool, leading to intermittent request failures even after the network stabilized.

Fault‑Drill Promotion

Recognizing the value of systematic fault practice, the team adopted chaos engineering principles to embed fault‑drill routines into their stability workflow.

Chaos Engineering Overview

Chaos engineering experimentally disrupts systems to assess resilience. The team uses an internal platform based on ChaosBlade, enabling controlled fault injection such as network loss, latency, and service termination.

Fault‑Drill Principles

Clarity: Define precise fault scenarios before execution.

Control: Minimize blast radius by estimating impact and containing faults.

Preparation: Design emergency handling plans for uncontrolled outcomes.

Interruptibility: Ensure faults can be stopped at any time.

Execution Workflow

Pre‑conditions: Ensure observability tooling (logs, monitoring, APM, traffic shaping) is in place.

Preparation Phase: Confirm scenario, impact scope, observation metrics, emergency plan, collaboration plan, and conduct a review.

Execution Phase: Inject faults according to the plan, monitor metrics, and keep on‑call personnel ready.

Recovery Phase: Verify fault cessation, continue monitoring to confirm stability, and ensure all injected faults are fully disabled.

Review Phase: Document actions, outcomes, and lessons; identify improvements across architecture, tooling, processes, and people.

Continuous Cycle: Institutionalize regular fault‑drill cycles to maintain readiness.

Vision

The team aims to integrate fault drills with CI/CD, automate execution and monitoring, randomize scenarios, and link drills with APM, precise testing, and service‑dependency data to achieve accurate, low‑risk fault simulations.

In summary, the network jitter incident exposed a Redisson connection‑pool bug; systematic investigation, reproducible testing, and a structured fault‑drill framework turned a painful outage into a learning opportunity for improving system reliability.