Fundamental Methods for Service Troubleshooting and Redis Performance Optimization

The author describes a real‑world incident where an alert signaled that overall data processing time had suddenly increased, prompting an immediate investigation.

1. Service‑level Diagnosis

The first step is to pinpoint which module and which stage are slow. By checking basic resource metrics—memory, CPU, node load, and disk usage—the author shows that all these indicators were normal, suggesting the issue lies elsewhere.

Two possible causes are considered:

Problems within the module itself.

Issues related to data volume.

Since the module had not been recently deployed, the focus shifts to data volume, which had grown five‑fold. The immediate mitigation applied was scaling, rate limiting, and service degradation, which resolved the alert.

2. Redis‑specific Diagnosis

Because the slowdown was traced to Redis response times, a deep dive into Redis health is performed.

Key checks include:

Network latency between the application server and Redis ( redis-cli -h 127.0.0.1 -p 6379 --intrinsic-latency 60 and redis-cli -h 127.0.0.1 -p 6379 --latency-history -i 1 ).

Throughput via INFO STATS (commands processed, ops/sec, network I/O).

Memory usage and fragmentation ( used_memory_rss_human , mem_fragmentation_ratio ).

Replication status ( INFO REPLICATION ).

Key count ( INFO KEYSPACE ).

Potential big keys and their impact.

Slowlog configuration and analysis.

Hotkey detection (requires LFU eviction policy).

Profiling with pprof reveals that Redis commands related to a specific function dominate the latency, confirming Redis as the bottleneck.

3. Reproducing and Testing the Issue

The author reproduces the problem locally using pipelines and Lua scripts, then simulates load on the production pipeline with Kafka‑based traffic generators ( cat xxx-test | kaf produce kv__0.111 -n 10000 -b broker:9092 ), gradually increasing pressure until the Redis CPU usage spikes.

Observations show that hotkeys cause high CPU consumption while overall OPS remain modest, indicating that the single‑threaded Redis model is saturated by a few hot operations.

4. Mitigation Strategies

Three practical remedies are proposed:

For multi‑instance deployments, employ read/write splitting.

For single‑instance setups, use command pipelines to batch writes.

If pipelines are insufficient, introduce an additional caching layer in front of Redis.

Applying a cache layer reduced service latency and lowered Redis CPU usage dramatically.

In summary, the article demonstrates a systematic approach to service and Redis troubleshooting: start with high‑level resource checks, narrow down to data‑volume effects, perform detailed Redis diagnostics, reproduce the issue under controlled load, and finally apply scaling, pipelining, or caching to restore performance.