Redis Memory Crisis: Big Key Detection & Expired Data Governance

Phenomenon & Background

At 2 AM, monitoring alerts show P99 latency spiking from 5ms to 800ms+, with used_memory nearing maxmemory. Logs reveal OOM command not allowed when used memory > 'maxmemory' errors, rejecting writes. Typical symptoms: memory keeps rising, key count grows slowly or not at all, individual commands take unusually long, accompanied by slow queries or replication lag.

The root cause is usually not Redis itself but data model and lifecycle management: a Hash or ZSet growing without bounds, caches missing TTL, or large objects serialized into a single String. The solution is not to increase maxmemory but to find the memory‑hogging keys, understand why they are large and not reclaimed, and fix the source.

Target audience: intermediate ops and DevOps engineers. Assumed environment: Redis 6.2 or 7.x (standalone or master‑replica; Cluster notes included), monitoring via redis_exporter/Prometheus/Grafana, standard client access, read‑only or bastion access to redis‑cli. Production forbids blocking commands like KEYS * or FLUSHALL.

Core Knowledge

Redis Memory Composition

INFO memory

fields must be distinguished: used_memory: allocator’s view of actual memory used (data, client buffers, replication buffers, Lua cache). Basis for maxmemory. used_memory_rss: OS‑level RSS. Difference from used_memory is mainly fragmentation. mem_fragmentation_ratio = used_memory_rss / used_memory. Normal 1.0–1.5; >1.5 indicates severe fragmentation; <1.0 means swap (dangerous). used_memory_dataset: memory for data only, excluding client buffers. mem_clients_normal, mem_clients_slaves: client output buffers and replica buffers. Spikes often mean slow consumers or massive Pub/Sub.

Common mistake: judging data growth by used_memory_rss alone. Must also check used_memory, fragmentation ratio, and mem_clients_*.

What Is a Big Key

No official definition; engineering thresholds:

Single key > 1 MB (common heuristic, depends on P99 latency requirements).

Collection elements > 5,000 (Hash fields, Set members, ZSet members, List elements).

String value > 100 KB (e.g., full JSON or Base64 image).

Three harms:

Deletion blocking: DEL on a million‑element collection blocks the main thread for hundreds of ms.

Read/write amplification: HGETALL, SMEMBERS, LRANGE 0 -1 return huge payloads, consuming bandwidth and blocking the event loop.

Replication/migration pressure: full sync and Cluster slot migration suffer from bloated replication buffers and timeouts.

Expiration Mechanism

Lazy expiration: check on access, delete if expired.

Active expiration: background periodic sampling of keys with TTL. Default 10 times/sec (controlled by hz, default 10). Each round samples a batch; if expired ratio high, continues looping.

Implications: keys without TTL are never auto‑reclaimed. Many keys expiring simultaneously cause CPU spikes and latency jitter — add random jitter to TTL to avoid avalanche.

Eviction Policies

When used_memory hits maxmemory, maxmemory-policy decides: noeviction (default): writes error out; for storage that cannot lose data. volatile-lru, volatile-lfu, volatile-ttl, volatile-random: evict only keys with TTL. allkeys-lru, allkeys-lfu, allkeys-random: evict any key; suitable for pure cache.

Eviction is a last resort and loses data. If business depends on certain keys, solve via capacity and lifecycle, not eviction. volatile-* policies effectively become no‑eviction if most keys lack TTL, leading to write failures.

Overall Approach: Stop Bleeding → Locate → Govern

Stop bleeding : restore writes without new risks. Temporary memory scale‑up, degrade non‑critical writes, targeted cleanup of disposable cache keys. Avoid blindly switching to allkeys-lru unless instance is pure cache.

Locate : find top memory‑consuming keys, their types, sizes, TTL distribution. Tools: redis-cli --bigkeys, --memkeys, custom SCAN scripts, MEMORY USAGE for precise measurement.

Govern : control growth at source. Add TTL, split big keys, redesign data structures, add cleanup jobs, adjust client write logic.

Practical Steps

Step 1: Confirm Instance State & Memory Level

Use REDISCLI_AUTH env var for password (avoids process list/history). Check connectivity and version:

export REDISCLI_AUTH="$(cat /etc/redis/.auth_prod_cache)"
REDIS_HOST="10.0.2.21"
REDIS_PORT="6379"
redis-cli -h "$REDIS_HOST" -p "$REDIS_PORT" PING
redis-cli -h "$REDIS_HOST" -p "$REDIS_PORT" INFO server | grep -E "redis_version|redis_mode|uptime_in_days"

Then examine memory metrics:

redis-cli -h "$REDIS_HOST" -p "$REDIS_PORT" INFO memory | grep -E "^used_memory_human|^used_memory_rss_human|^used_memory_peak_human|^maxmemory_human|^maxmemory_policy|^mem_fragmentation_ratio|^mem_clients_normal|^mem_clients_slaves"

Example output shows used_memory_human:7.82G, maxmemory_human:8.00G, maxmemory_policy:noeviction, mem_fragmentation_ratio:1.04, mem_clients_normal:18MB. Interpretation: memory near limit with noeviction → emergency; fragmentation normal → growth is data; client buffers small → not the cause.

Step 2: Key Count & TTL Distribution

INFO keyspace

reveals keys=12841093, expires=3920411 → ~69% keys have no TTL (classic unbounded growth signal). avg_ttl ~790 hours, far beyond expected cache lifetime, indicating some keys have excessively long TTL.

Step 3: Bigkeys Scan

redis-cli --bigkeys -i 0.1

(throttle with -i) samples all keys, reporting largest per type. Output shows biggest hash user:session:index with 1,820,345 fields, biggest string report:cache:20260901 ~4.3 MB, biggest zset rank:global:daily ~980k members. Note: --bigkeys only shows the single largest per type; many large keys may exist. Many empty lists/hashes indicate missing cleanup logic.

For top‑N by memory, use --memkeys (Redis 7.x). If unavailable, fall back to custom SCAN script.

Step 4: Precise Location with SCAN + MEMORY USAGE

MEMORY USAGE key [SAMPLES count]

returns actual bytes (includes overhead). For large collections, use SAMPLES to reduce cost. Provided bash script ( redis_find_bigkeys.sh) scans with SCAN COUNT 500, calls MEMORY USAGE and TYPE per key, outputs TSV of keys exceeding threshold (e.g., 1 MB). Runs read‑only; best on replica, rate‑limited with sleep 0.05. Example run found 1,384 keys >1 MB; top entries: hash user:session:index 1.8 GB, string report:cache:20260901 4.3 MB. Pattern: same prefix → systemic issue; single key dominating → primary culprit.

Step 5: Analyze Big Key Access Pattern & Source

Inspect user:session:index:

TYPE → hash
OBJECT ENCODING → hashtable (exceeded listpack/ziplist threshold)
HLEN → 1,820,345
TTL → -1 (no expiry)
MEMORY USAGE → 1,820,345,040 bytes (~1000 bytes/field)

HSCAN

sampling shows fields like session:idx:u_8831230 with session IDs → reverse index from user to session, but expired sessions never removed. CLIENT LIST and short MONITOR (5 sec) reveal writes from IP 10.0.3.41 via HSET only — single service writing without deletion logic.

Step 6: Confirm Root Cause

Evidence table correlates multiple signals: INFO memory: used_memory near maxmemory, fragmentation 1.04 → growth is data, not fragmentation. INFO keyspace: 69% keys lack TTL → massive non‑expiring data.

Bigkeys/script: user:session:index 1.8 GB → single hash is main contradiction.

TTL=-1, 1.82M fields → index only grows.

HSCAN: fields map to expired sessions → business fails to clean index.

MONITOR: writes only from 10.0.3.41 → single service responsible.

Only combined evidence yields reliable root cause: session index lacks expiration, causing unbounded Hash growth.

Step 7: Remediation Plan

Short‑term: batched deletion. Avoid DEL blocking; use UNLINK (background free) for whole keys, or batched HDEL for partial cleanup. Pseudocode:

cursor = 0
loop:
  (cursor, fields) = HSCAN key cursor COUNT 500
  expired_fields = filter(fields, session_expired)
  if expired_fields not empty: HDEL key expired_fields...
  sleep 50ms
  if cursor == 0: break

Real implementation: limit HDEL to ≤500 fields, sleep between batches, dry‑run first, confirm deletion criteria with developers. If entire key is disposable cache, UNLINK directly.

Long‑term: write‑side governance.

Set TTL on all cache keys with random jitter (e.g., 7 days + random(0,3600) seconds).

Add cleanup for index data: delete index entries when session expires, or run periodic cleanup job.

Shard large collections: split user:session:index by user ID hash into 256 keys.

Redis 7.4+ supports Hash field‑level expiration ( HEXPIRE, HPEXPIRE) for automatic per‑field expiry without application scans.

Replica Memory Higher Than Master

After fixing big keys, replica used_memory may stay higher. Common causes: replica’s client output buffers, replication buffers during full sync, differing fragmentation. Troubleshooting steps:

Compare INFO memory and INFO replication on both sides. Example: replica mem_clients_slaves 1.5 GB (cascade replication), master 0; offsets differ by only 4.4 KB → replication nearly caught up; fragmentation equal.

Check if replica is also a master for downstream ( connected_slaves > 0). If downstream lag grows, fix downstream network/load.

Examine

CLIENT LIST

omem

(output buffer) to find slow consumers. Example: 1.5 GB buffer on a psync connection → downstream replica lagging.

Remediation: fix downstream, adjust client-output-buffer-limit replica, or carefully CLIENT KILL after verification.

Command Cheatsheet

Key commands grouped: connection/auth, memory/keyspace overview, big key scans ( --bigkeys, --memkeys), single key inspection ( TYPE, OBJECT ENCODING, MEMORY USAGE, TTL), slowlog, config get/set/rewrite, deletion ( UNLINK preferred over DEL for large keys). Note: UNLINK introduced in Redis 4.0; older versions must use batched DEL.

Configuration Examples

Reduce Memory Overhead for Small Collections (Redis 7.x)

hash-max-listpack-entries 128
hash-max-listpack-value 64
slowlog-log-slower-than 10000
slowlog-max-len 512
maxmemory-policy volatile-lru

Apply at runtime with CONFIG SET and persist with CONFIG REWRITE. Parameters affect only new writes; existing large hashes stay in hashtable encoding.

Client Output Buffer Limits

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

replica

limit prevents slow replicas from ballooning buffer; normal default unlimited — set if slow query clients exist.

Logs & Metrics

Redis Log Keywords

Grep for OOM, maxmemory, evict, fork, COW, Background saving, Asynchronous AOF, Connection reset. Typical entries: OOM error confirms write rejection; fork start indicates RDB snapshot (COW memory spike); AOF fsync slow points to disk I/O bottleneck.

Key Metrics (redis_exporter)

Important metrics with alert thresholds: redis_memory_used_bytes / redis_memory_max_bytes > 80% (used_memory). redis_memory_used_rss_bytes > 90% maxmemory (RSS). redis_mem_fragmentation_ratio > 1.5 or < 1.0 (swap emergency). redis_db_keys_expiring / redis_db_keys < 30% (too many non‑TTL keys). redis_evicted_keys_total rate > 0 (eviction active). redis_expired_keys_total drop (expiration anomaly). redis_connected_clients spike (connection leak). redis_blocked_clients > 0 sustained (BLPOP or big key ops). redis_slowlog_length rising (slow queries).

PromQL examples: memory usage %, non‑TTL ratio, 5‑min eviction rate. Set thresholds based on historical baselines.

Investigation Path

Standard ordered checklist:

Confirm policy & water level ( INFO memory).

Rule out fragmentation & COW ( mem_fragmentation_ratio, INFO persistence).

Rule out client buffers ( mem_clients_*, CLIENT LIST omem).

Key count & TTL ( INFO keyspace).

Locate big keys ( --bigkeys, SCAN script).

Analyze write source ( CLIENT LIST, short MONITOR).

Confirm root cause & fix (combine logs & code).

Verify & postmortem (trends, alerts, business metrics).

Risks & Rollback

High‑Risk Operations

Table summarizing operation, impact, risk, alternative, rollback:

DEL/UNLINK big key : data loss, DEL blocks → batched HDEL, UNLINK; restore from backup or rebuild cache.

CONFIG SET maxmemory-policy : accidental eviction → scale memory instead; revert policy.

CONFIG SET maxmemory : swap/OOM Killer → scale instance; revert value.

Master‑replica failover : write pause, inconsistency → low‑peak, verify offsets; fail back after data check.

FLUSHDB/FLUSHALL : total data loss → never in prod; restore from RDB/AOF.

Modify hash-max-listpack-entries : limited effect (new writes only) → low risk; revert.

Pre‑Deletion Checks

Confirm key nature: pure cache vs. sole data source.

Confirm owner: get developer sign‑off.

Backup: DUMP key (binary, use base64) or full --rdb snapshot; verify with redis-check-rdb.

Rollback Plans

Mis‑delete: restore key from RDB via temporary instance + RESTORE.

Wrong eviction policy: revert to noeviction, assess lost data.

Failover issues: fail back to original master, compare master_repl_offset.

Production Best Practices

Execution window & rate limiting : big key scans/deletes only in low‑peak (e.g., 01:00–05:00), notify teams. Throttle with -i and batch sleeps; monitor command latency/P99.

Permissions & audit : use Redis 6+ ACL for read‑only ops accounts, disable dangerous commands ( DEL, UNLINK, FLUSHALL, CONFIG). Persist ACL via aclfile or ACL SAVE. Log all manual changes in change tickets.

Prefer replica for scans : run SCAN on replicas (read‑only by default) to avoid master load.

Replication & persistence impact : big deletions generate replication traffic; verify master_repl_offset vs slave_repl_offset before deleting. AOF grows; periodic rewrite compacts.

Capacity planning : compute daily growth rate, steady‑state size under TTL policy, keep >20% headroom. If steady state exceeds single instance, shard (Redis Cluster) or split business instances.

Cluster specifics : --bigkeys / SCAN per master node; connect directly to each node address, not via -c redirection.

Verification

Post‑remediation confirmation:

Memory trend: used_memory drops after deletion, stays flat for a week.

Key structure: keys / expires ratio returns to baseline, non‑TTL keys stable.

Big key count: re‑run script, threshold‑exceeding keys significantly reduced.

Business metrics: P99 latency back to baseline, error rate down, no new OOM logs.

Fragmentation: if high, try MEMORY PURGE (jemalloc only).

Postmortem Template

Record: event name, time window, impact scope, root cause, detection method, stop‑gap measures, fix actions, long‑term improvements, open issues.

Summary

Redis high memory is a symptom; the essence is missing data lifecycle management. Key takeaways:

Use INFO memory and INFO keyspace to distinguish data growth, fragmentation, or client buffers.

Locate big keys with --bigkeys and SCAN+ MEMORY USAGE for quantification.

Combine TTL, HLEN, sampling, and write source to confirm root cause — not size alone.

Delete safely: UNLINK or batched HDEL; backup and verify data nature first.

Long‑term fix at write side: TTL with jitter, synchronous index cleanup, shard large collections.

Continuous verification via monitoring; add non‑TTL ratio and big key count to daily inspections.

Eviction policies and scaling only buy time; eliminating unbounded growth at the data model level closes the issue for good.