Diagnosing High Memory Usage in a Redis Cluster Using Memory Distribution Monitoring and Bigkeys

Background: In a production Redis 5.0.10 cluster with over 30 shards, one shard showed abnormal memory usage exceeding 70% (≈356 MB of 512 MB). The article demonstrates how to investigate this issue.

Diagnosis – Memory usage distribution monitoring: The problematic shard used about 356 MB while other shards stayed below 100 MB.

Comparison of abnormal and normal instances: The abnormal instance stored fewer keys (≈191 k vs 637 k) but its objects consumed roughly twice the memory.

### 正常实例
redis-cli -p 6380 -h 10.186.62.28 info keyspace   # data volume
db0:keys=637147,expires=0,avg_ttl=0
redis-cli -p 6380 -h 10.186.62.28 info memory |grep -w used_memory
used_memory:104917416

### 异常实例
redis-cli -p 6382 -h 10.186.62.5 info keyspace   # data volume
db0:keys=191433,expires=0,avg_ttl=0
redis-cli -p 6382 -h 10.186.62.56 info memory |grep -w used_memory
used_memory:373672656

Fragmentation ratio: The abnormal instance showed a fragmentation ratio of 0.89, indicating fragmentation was not the cause.

redis-cli -p 6382 -h 10.186.62.56 info memory |grep mem_fragmentation_ratio
mem_fragmentation_ratio:0.89  ## 碎片率小于1

Bigkeys scanning analysis: Using redis-cli --bigkeys during a low‑traffic window identified two large string keys—“bigk:0” (~200 MB) and “bigkkkkk:0” (~100 MB)—which together accounted for most of the memory spike.

# redis-cli -p 6382 -h 10.186.62.56  --bigkeys

[...output showing biggest strings...]

Sampled 191433 keys in the keyspace!
Total key length in bytes is 4161149 (avg len 21.74)
Biggest string found "bigk:0" has 204800000 bytes
...

Memory usage of the identified keys confirmed the consumption:

10.186.62.56:6382> memory usage bigkkkkk:0
(integer) 117440568
10.186.62.56:6382> memory usage bigk:0
(integer) 234881072

Conclusion: When memory distribution is uneven in a Redis cluster, running redis-cli --bigkeys provides a quick way to locate oversized keys, but it should be executed during off‑peak periods. Additional debug or memory commands can be used to generate large keys or simulate blocking for further testing.