Redis Memory Growth, Eviction Policies & Large Key Governance: A Complete Operational Guide

1. Problem Background: A Typical "Slow Memory Rise"

An 8-instance product detail cache cluster, each with maxmemory 16gb and maxmemory-policy allkeys-lru, ran stably for six months. Then alerts fired: three instances saw used_memory creep from 9GB to 15.5GB, used_memory_rss neared 17GB, and the host began swapping. The on-call fix was raising maxmemory to 18GB — a painkiller. Two weeks later the same issue returned. Investigation revealed three facts: used_memory vs used_memory_rss gap was 1.3GB, but mem_fragmentation_ratio stayed at 1.08 — not fragmentation. INFO stats showed evicted_keys growing only a few hundred per day — eviction barely triggered because memory was expanded before hitting maxmemory. --bigkeys scan found a Hash with 1.2M fields and a List with 900K elements; MEMORY USAGE confirmed the Hash consumed 1.3GB.

Root cause: the business wrote a "per-SKU accumulated behavior log" into a single Hash via HSET sku:behavior with no TTL. The "memory growth" was fundamentally a data-model design flaw; maxmemory is merely the last safety net.

2. Core Knowledge: Where Memory Goes

2.1 Three-Layer Memory View

INFO memory

key fields: used_memory: Redis's own accounting via zmalloc — data structures, object overhead, replication buffers, client buffers. used_memory_rss: OS resident set size = used_memory + allocator fragmentation (allocated but unreleased pages) + code segment + stacks. used_memory_dataset = used_memory - used_memory_overhead; pure data payload. Overhead includes main dict, expires dict, client output buffers.

Interpretation rules:

rss/used_memory 1.0–1.1 : Normal — no action.

Ratio 1.2–1.5 : Acceptable fragmentation — watch trend, avoid frequent add/delete.

Ratio > 1.5 : Severe fragmentation, usually from mass deletes — enable activedefrag.

rss < used_memory : Swapping occurred — Emergency — check swap & memory overcommit.

used_memory_peak far above current : Historical spike — investigate past large keys or bulk writes. rss < used_memory means the OS swapped Redis pages to disk — a single swap can push P99 from 1ms to 500ms. Common in containers where the container memory limit is set below Redis maxmemory or the host overcommits memory. Verify with free -m and cgroup memory.max / memory.current. Rule of thumb: set maxmemory to 70–75% of memory.max to leave room for fragmentation, replication buffers, and fork COW.

2.2 Data Structure Memory Overhead

Each type has a "small-object encoding" and a "large-object encoding"; crossing the threshold jumps memory usage. Critical for diagnosing "why 1M keys take 200MB".

String : small encoding int / embstr (<= 44 bytes for embstr); large encoding raw (SDS).

List : small encoding listpack ( list-max-listpack-size 128, element <= 64 bytes); large encoding quicklist (multi-node).

Hash : small encoding listpack ( hash-max-listpack-entries 128, hash-max-listpack-value 64); large encoding hashtable.

Set : small encoding intset / listpack ( set-max-intset-entries 512, set-max-listpack-entries 128); large encoding hashtable.

ZSet : small encoding listpack ( zset-max-listpack-entries 128, zset-max-listpack-value 64); large encoding skiplist + dict.

Example: Hash with <=128 fields uses listpack (~20–30 bytes/field). Beyond 128 fields or any value >64 bytes, it converts to hashtable — each entry stores key pointer, value pointer, and dictEntry; jemalloc size-class alignment pushes per-field overhead to 80–100 bytes. The same data can triple in memory after crossing the threshold. Hence "stuffing 10K user states into one Hash" may cost more than 10K separate Strings once fields exceed 128. Check encoding with OBJECT ENCODING key.

2.3 SDS & Object Header Overhead

String memory = SDS header (length-dependent) + data + + robj struct + jemalloc size-class alignment. jemalloc allocates in bins: 8, 16, 32, 48, 64, 80, 96, 112, 128… so a 45-byte string may occupy 64 bytes. Capacity estimation formula:

single key memory ≈ (key_len + val_len + 56) × 1.15

56 covers robj, SDS header, dictEntry, alignment. For 1M keys with 32-byte key + 128-byte value: (32+128+56)×1.15 ≈ 248 bytes → ~248MB. Actual used_memory often 260–280MB due to expires dict, replication buffers, etc. Never use "key count × average size" for capacity planning — it underestimates by 30%+.

2.4 Expires Dictionary Overhead

Every key with TTL exists in both redisDb.dict and redisDb.expires. Each expires entry costs ~32–48 bytes. 5M keys with TTL adds 160–240MB just for the expires dict — comparable to the data itself if values are tiny. Setting TTL on all keys is usually worth it to prevent unbounded growth, but avoid "set TTL then never read" — only active expiry cleans those, which is inefficient.

2.5 Two Expiry Deletion Paths

Lazy deletion : check on access; cheap but cold data never removed.

Active expiry ( activeExpireCycle): runs several times per second, random-samples expires dict, deletes expired keys; if >25% expired in a round, continues. Intensity controlled by active-expire-effort (default 1, range 1–10).

Monitor expired_stale_perc (sampled expired-but-not-deleted ratio) and expired_time_cap_reached_count (times active expiry was throttled). Persistent growth in the latter means active-expire-effort too low or CPU starved.

2.6 Fragmentation Sources

Data size distribution changes (e.g., 1KB → 200B values leave unreleasable 1KB pages).

Mass deletes — freed pages can't be reused across size classes.

jemalloc arena strategy — threads allocate in different arenas; one arena's free pages can't satisfy another's request.

If mem_fragmentation_ratio > 1.4 long-term and RSS grows, enable active defrag:

CONFIG SET activedefrag yes
CONFIG SET active-defrag-ignore-bytes 100mb
CONFIG SET active-defrag-threshold-lower 10
CONFIG SET active-defrag-threshold-upper 100
CONFIG SET active-defrag-cycle-min 1
CONFIG SET active-defrag-cycle-max 25

activedefrag

runs in the main thread, consuming CPU. cycle-max 25 caps at 25% CPU; higher values hurt command latency. Only works with jemalloc (verify via INFO memory | grep mem_allocator).

3. Eviction Policies: Eight Strategies & Boundaries

3.1 Policy Comparison

noeviction : scope none; algorithm error on write; use case config center, queues, distributed locks; avoid when cache.

allkeys-lru : scope all keys; algorithm approx LRU; use case general cache; avoid when hotset needs protection.

allkeys-lfu : scope all keys; algorithm approx LFU; use case clear hotspots (e-commerce detail, homepage config); avoid when uniform access pattern.

allkeys-random : scope all keys; algorithm random; use case rare; avoid —.

volatile-lru : scope keys with TTL; algorithm approx LRU; use case mixed cache + persistent data on same instance; avoid when some keys lack TTL.

volatile-lfu : scope keys with TTL; algorithm approx LFU; use case same as above with clear hotspots; avoid when some keys lack TTL.

volatile-random : scope keys with TTL; algorithm random; use case rare; avoid —.

volatile-ttl : scope keys with TTL; algorithm shortest remaining TTL first; use case explicit "earlier expiry = delete first" semantics; avoid most caches.

Selection rule: Pure cache → allkeys-* ; mixed → volatile-* . Hidden trap: if zero keys have TTL , volatile-* degrades to noeviction — writes fail with OOM command not allowed when used memory > 'maxmemory'. This explodes when a team adds non-TTL keys.

3.2 LRU Is Approximate

Redis doesn't maintain a true LRU list (too much pointer overhead). Instead, each redisObject stores an access timestamp. On eviction, it samples maxmemory-samples keys (default 5) and evicts the oldest. Implications:

Smaller sample → closer to random eviction.

Default 5 gives ≤10% hit-rate gap vs true LRU.

Raising to 10 improves accuracy but costs more CPU per eviction. CONFIG SET maxmemory-samples 10 Counter-intuitive: newly written keys can be evicted immediately because timestamps have only second-level precision; batch writes exacerbate this. Mitigation: read after write (refresh timestamp) or switch to LFU.

3.3 LFU Counter Details

LFU uses an 8-bit counter + 8-bit decay time per object. Counter increments probabilistically: higher counter → lower increment probability (logarithmic growth), preventing hot keys from saturating the counter.

CONFIG SET lfu-log-factor 10   # controls growth difficulty (default 10)
CONFIG SET lfu-decay-time 1    # decay interval in minutes (default 1)

Too small lfu-decay-time → yesterday's hot keys lose weight too fast; too large (e.g., 1440) → stale hot keys block new ones. View a key's LFU counter: OBJECT FREQ key (only works under *-lfu policies).

3.4 Why Eviction "Appears Ineffective"

maxmemory set above actual available memory. Machine has 16GB, maxmemory 16gb, but used_memory + fragmentation + replication buffers + fork COW already exceed 16GB — OS OOM kills before Redis evicts. Fix: maxmemory ≤ 70% physical RAM; ≤60% if RDB forks.

maxmemory-policy is noeviction . Someone changed it at runtime but forgot to persist; restart reverts to default noeviction. Verify with CONFIG GET maxmemory-policy.

Memory growth in "non-data" areas. Replication backlog ( repl-backlog-size), client output buffers ( client-output-buffer-limit) count toward used_memory but are unevictable. Real case: replica's 512MB backlog + 14GB data > maxmemory 14gb → replica kept evicting yet stayed over limit.

3.5 Eviction Performance Impact & Lazyfree

Eviction runs on the main thread; evicting a large key (e.g., 1M-field Hash) blocks for hundreds of ms to seconds. Solution: enable lazyfree for async release:

CONFIG SET lazyfree-lazy-eviction yes
CONFIG SET lazyfree-lazy-expire yes
CONFIG SET lazyfree-lazy-server-del yes
CONFIG SET lazyfree-lazy-user-del yes

lazyfree-lazy-eviction

: async free on eviction. lazyfree-lazy-expire: async free on expiry. lazyfree-lazy-server-del: async free on internal deletes ( RENAME, SUNIONSTORE overwrite). lazyfree-lazy-user-del: DEL becomes UNLINK.

Caution: lazyfree delays memory reclaim; lazyfree_pending_objects shows backlog. If it stays non-zero, background thread can't keep up. Recommended: enable the last three freely; enable lazyfree-lazy-eviction only when keys are large and brief over-limit is acceptable.

3.6 Replication & Client Buffers

These don't participate in eviction but consume used_memory — common cause of "eviction configured but memory doesn't drop".

Replication backlog ( repl-backlog-size, default 1MB): enables partial resync (PSYNC). Too small → frequent full syncs (fork + RDB + network). Size by peak write rate × tolerated disconnect time. E.g., 50MB/s × 30s = 1.5GB — non-trivial.

Client output buffers ( client-output-buffer-limit): per-class hard/soft limits. Replica hard limit 256MB; exceeding disconnects replica. Often caused by large key writes or slow replica consumption. Monitor client_recent_max_output_buffer.

4. Large Keys: Definition, Hazards & Discovery

4.1 What Counts as Large

Element count : String > 10KB; collections > 5K elements — good for app-side validation.

Memory footprint : Single key > 1MB (cache) / 10MB (wide table) — use MEMORY USAGE for detection.

Practical rule: if a single command on a key exceeds 5ms, or its presence pushes replication lag >1s, it's a large key regardless of size.

4.2 Four Harms

Command blocking : HGETALL on 1M-field Hash takes hundreds of ms; all other requests queue.

Network saturation : 10MB value at 100 QPS = 1GB/s — NIC saturates; symptom: "Redis CPU low but clients time out".

Deletion blocking : DEL on large key synchronously frees memory, blocking main thread for seconds.

Replication lag & data skew : large key propagates to all replicas; in Cluster, its slot becomes heavier, slowing resharding and causing imbalance.

4.3 Discovery Methods Compared

redis-cli --bigkeys : SCAN + TYPE + length cmds; simple, no extra deploy; only element count, not real memory.

redis-cli --memkeys : SCAN + MEMORY USAGE; real memory; slow; sampling error on large keys.

redis-cli --hotkeys : SCAN + OBJECT FREQ; finds hotspots; requires *-lfu policy.

SCAN + custom script : controllable; filter by prefix, threshold; dev/maintenance effort.

Offline RDB analysis : parse RDB file; zero online impact; needs RDB file; stale.

Slowlog : logs commands > threshold; directly shows slow commands; only command latency, not memory.

4.4 Proper --bigkeys Usage

redis-cli -h host -p 6379 -a "$PASS" --no-auth-warning --bigkeys -i 0.05

-i 0.05

sleeps 50ms per 100 keys — mandatory in production to avoid CPU/latency spikes. Output shows only the largest key per type, not all large keys. String bytes = STRLEN (no object overhead); collection numbers = element count, not bytes. 5K-field Hash with 1KB values = 5MB; with 10B values = 50KB. Full scan on 8M keys takes minutes even with -i.

4.5 Precise Location with SCAN + MEMORY USAGE

redis-cli -h host -a "$PASS" --no-auth-warning MEMORY USAGE sku:behavior
# 1384120320 → ~1.32GB (includes key, value, encoding overhead)

More accurate than --bigkeys element counts. Production-grade scan script (rate-limited, canary switch) provided in article — scans 8M keys in 40–90 mins. Note: MEMORY USAGE defaults to sampling; MEMORY USAGE key SAMPLES 0 forces full traversal for exact size (slow).

4.6 Slowlog for Large Key Detection

SLOWLOG

logs commands exceeding slowlog-log-slower-than (default 10ms). A 2M-field HGETALL taking 143ms appears in slowlog. Query:

redis-cli --json SLOWLOG GET 1024 | jq -r '.[] | [.[3][0], .[3][1] // "", .[2]] | @tsv' | sort -k3 -nr | head -20

Fields: log ID, timestamp, duration (µs), command+args, client addr, client name. Slowlog only measures command execution time, not network/queue latency — real user latency is higher. Recommended: set slowlog-max-len 1024, keep default 10ms threshold, export daily, aggregate by command+key prefix.

5. Remediation: Safe Deletion After Discovery

5.1 Classify Before Action

Structural error : Hash with 1.2M fields as wide table → split into multiple keys, shard by dimension; code change required.

Large cache without TTL : hundreds of K keys all persist → add TTL, let eviction handle; maybe code change.

Historical debris : old version writes, now unused → direct delete; no code change.

Queue backlog : List/Stream consumer stopped → fix consumer first, then delete; code change required.

Leaderboard/dedup set : ZSet/Set growing unbounded → add window, periodic ZREMRANGEBYRANK; code change required.

Verify "no longer used" with OBJECT IDLETIME key (seconds since last access; LRU clock) and short MONITOR sampling (≤10s, production caution). Both must indicate zero access.

5.2 Why UNLINK Not DEL

DEL

synchronously frees memory — 1.2M-field Hash blocks main thread 300ms–1s. UNLINK unlinks key from keyspace (O(1)), hands memory to background thread (lazyfree). Client returns immediately; main thread unblocked. Requires lazyfree params enabled (check CONFIG GET lazyfree-*). Note: lazyfree-lazy-user-del defaults to no (so DEL stays sync), but UNLINK is always async regardless. UNLINK doesn't solve read-blocking or network transfer; massive concurrent UNLINK queues background thread, indirectly affecting main thread allocation.

5.3 Progressive Deletion for Collections

Hash : HSCAN + batched HDEL. Script provided: scans with single HSCAN call (cursor + field/value pairs), extracts fields via awk 'NR>1 && NR%2==1', deletes one-by-one (safe for special chars), sleeps every 5K deletions, supports MAX_DEL for canary runs.

List : LPOP key count (Redis 6.2+) — pops 1000 at a time, far faster than looped single pops.

Set : SSCAN + SREM (same space/tab caveat).

ZSet : ZREMRANGEBYSCORE key -inf timestamp LIMIT 0 500 (Redis 6.2+ supports LIMIT; older versions use ZREMRANGEBYRANK).

Stream : XTRIM mystream MAXLEN ~ 100000 (approximate, macro-node delete, efficient) or MINID for exact ID-based trim. Post-delete memory may not drop immediately due to radix tree nodes not shrinking; UNLINK + rebuild is cleaner.

5.4 Pre-Deletion Checklist

Backup : For recoverable data, DUMP (small keys) or HSCAN batched export to JSON Lines (large keys). RESTORE requires same Redis version; use REPLACE to overwrite (dangerous — verify EXISTS first).

Impact scope : Grep codebase for key prefix; confirm no hidden writers. Delete without stopping source → key regrows in days.

Execution window : Low traffic (02:00–05:00), avoid RDB/AOF rewrite, full sync windows.

Monitoring & rollback : Watch used_memory, latency, connected_clients, evicted_keys, business error rate. Rollback: stop delete, restore from backup via batched HSET / SADD / RPUSH.

5.5 Execution & Verification

Canary: delete 10% first ( MAX_DEL=120000), observe 15 mins. Criteria: used_memory smoothly drops, business P99 stable, no new slowlog entries of same type. Monitoring script samples every 2s: used_memory, used_memory_rss, mem_fragmentation_ratio. Post-delete verification: EXISTS key = 0, DBSIZE delta matches, MEMORY STATS dataset.bytes drops.

6. Monitoring: Catch Issues Before Explosion

6.1 Essential INFO Fields

No exporter needed — INFO memory gives half the answer. Key fields: used_memory, used_memory_rss, used_memory_peak, used_memory_dataset, maxmemory, mem_fragmentation_ratio, mem_allocator. mem_fragmentation_ratio &lt; 1 is the most dangerous signal — indicates swap; all memory accesses hit disk I/O, latency jumps from µs to ms/s. Check free -m and /proc/<pid>/status | grep VmSwap. Fix: lower maxmemory or add RAM, then swapoff -a (needs enough free RAM to pull swapped pages back).

6.2 redis_exporter Metric Names (v1.5x)

Common metrics: redis_memory_used_bytes, redis_memory_used_rss_bytes, redis_memory_used_peak_bytes, redis_memory_used_dataset_bytes, redis_memory_max_bytes, redis_mem_fragmentation_ratio, redis_evicted_keys_total (counter — use rate()), redis_expired_keys_total, redis_keyspace_hits_total / misses_total, redis_db_keys (per-db), redis_db_keys_expiring, redis_connected_clients, redis_blocked_clients, redis_commands_duration_seconds_total (by cmd), redis_commands_total (by cmd), redis_rdb_last_bgsave_status, redis_aof_last_write_status, redis_replication_offset, redis_slowlog_length. Always verify actual metric names after deploy — versions differ (e.g., redis_memory_used_bytes vs redis_used_memory_bytes).

6.3 Four Actionable Alerts

Memory high : redis_memory_used_bytes / redis_memory_max_bytes > 0.85 for 10m. If maxmemory=0, separate alert redis_memory_max_bytes == 0.

Heavy eviction : rate(redis_evicted_keys_total[5m]) > 10 for 5m. Threshold varies by cache sensitivity.

Fragmentation high : redis_mem_fragmentation_ratio > 1.5 for 30m. Only act if RSS also near physical limit.

Replication broken : redis_connected_slaves < 1 for 2m (critical). Master-replica disconnect risks data loss on failover.

6.4 Logs Fill Metric Gaps

Metrics show "memory grew"; logs show "why". Watch for: eviction start logs, RDB save failures ( fork: Cannot allocate memory), AOF rewrite failures, client buffer limit disconnects. fork: Cannot allocate memory often stems from vm.overcommit_memory=0 — Redis docs recommend 1 (always allow overcommit). Set via sysctl -w vm.overcommit_memory=1 and persist.

7. Capacity Planning: How Much Headroom

7.1 From Data Volume to Actual Memory

Count business keys & avg size (code or sampling).

Sample 1000 keys in staging with MEMORY USAGE, avg × 1.2 safety factor.

Add replication backlog ( repl-backlog-size, default 1MB, prod 64–256MB).

Add client buffer peaks (set client-output-buffer-limit normal explicitly).

Add AOF buffer + RDB fork COW (write-heavy: estimate 50%).

Final maxmemory = machine RAM × 0.7 (not 0.9). Reserve 30% for fork COW, fragmentation, OS cache.

Real example: 6M keys, used_memory 12.9GB, dataset 12.1GB (6% overhead), RSS 14.2GB (frag 1.10). 32GB machine, maxmemory 16GB — reasonable: leaves 16GB for fork COW (usually few hundred MB, but burst writes need margin).

7.2 When to Shard

Not just memory %; watch three signals:

Single instance used_memory > 20GB : RDB fork / AOF rewrite slow, recovery time grows.

QPS near single-core limit : instantaneous_ops_per_sec sustained ~100K.

Uncontrolled keys > 1MB : bandwidth & blocking risk.

Sharding options: by business module (more instances), by key prefix hash (client routing), or Redis Cluster (horizontal scale, but no cross-slot MGET /transactions/Lua, SELECT only db0, KEYS single-node). Cluster constraints force app changes.

7.3 Vertical Scaling Pitfalls

After adding RAM, Redis doesn't auto-use it — maxmemory is static config. Must update redis.conf and run CONFIG SET maxmemory 24gb (immediate) + CONFIG REWRITE (persists to file). CONFIG REWRITE fails if config file unwritable; verify CONFIG GET dir and startup config path, backup first.

8. Offline Analysis: Inspect Without Touching Production

8.1 redis-rdb-tools (Python)

rdb -c memory /backup/dump.rdb --bytes 1048576 --largest 50 > /tmp/bigkeys.csv

CSV columns: database,type,key,size_in_bytes,encoding,num_elements,len_largest_element,expiry. --bytes 1MB filters small keys; --largest 50 for quick triage. Older tool versions may mis-handle Redis 7.x listpack encodings — upgrade tool, not RDB. JSON output also available for further processing.

8.2 rdbtools-rs / rdb-cli (Rust)

For RDB >10GB, Python parser is slow & memory-hungry. Rust version is an order of magnitude faster. Verify CLI flags with --help. Always cross-check findings with live MEMORY USAGE — tool's memory model may differ from running version.

8.3 TTL Distribution from RDB

rdb -c csv /backup/dump.rdb 2>/dev/null | awk -F, 'NR>1 { total++; if ($8 != "" && $8 != "-1") ttl++ } END { printf "total=%d with_ttl=%d (%.2f%%)
", total, ttl, ttl*100/total }'

If with_ttl < 50%, most keys lack expiry — memory will only grow under traffic; root fix is pushing business to add TTL, not adding RAM.

9. Special Deployment Differences

9.1 Master-Replica + Sentinel

Replica maxmemory: replicas default replica-ignore-maxmemory yes (since 4.0) to guarantee consistency — replica used_memory may exceed maxmemory; don't misread as config failure.

Replication backlog: default 1MB too small for write-heavy or brief disconnects → full resync (fork + RDB + network). Set ≥64MB.

Failover memory spike: new master handles writes + full sync simultaneously; avoid BGSAVE during failover window, ensure new master has same RAM headroom.

9.2 Redis Cluster

Data skew : 16384 slots, but key distribution depends on business. One large key makes its node heavier. Check per-node used_memory + DBSIZE; use CLUSTER COUNTKEYSINSLOT / GETKEYSINSLOT to locate heavy slot.

No cross-slot ops : MGET / MSET /transactions/Lua require keys in same slot — use hash tags sku:{10086}:detail + sku:{10086}:behavior. Tag only uses first {...} content.

Migration consumes memory : MIGRATE serializes key; large keys block target node. Redis 8.4 CLUSTER MIGRATION (atomic slot migration, sync/async) is better but needs ≥8.4. Otherwise redis-cli --cluster reshard in low traffic.

Disabled commands : KEYS / SCAN single-node only; SELECT db0 only; FLUSHALL cluster-wide (7.0+ needs ASYNC for non-blocking).

9.3 Cloud-Managed Instances

No CONFIG SET — params via console, may trigger restart. INFO fields may be trimmed; some metrics only via cloud monitoring.

Use vendor's built-in "large key analysis" — safer than custom scripts.

Memory watermark alerts & write throttling behavior defined by platform; confirm beforehand.

Before deleting large keys, verify impact on billing/backup/read-replicas and check for "delete protection".

10. End-to-End Memory Rise Investigation

Real case: e-commerce cache instance memory 8GB → 15.8GB (limit 16GB), 85% watermark alert, intermittent cache misses.

10.1 Initial Triage (5 mins)

Four usual causes: write volume growth, keys without TTL, new/expanding large keys, buffer bloat (replication, client output, AOF). Quick commands: INFO memory (total, peak, RSS, frag), DBSIZE, INFO stats (evicted/expired/hits/misses), CLIENT LIST top output buffers.

10.2 Key Findings

used_memory:16965120000
used_memory_peak:16965120000  # now = historical peak, still rising
used_memory_rss:17825792000
maxmemory:17179869184
mem_fragmentation_ratio:1.05  # normal
used_memory == peak → still growing
rss/used ≈ 1.05 → not fragmentation
evicted_keys:0 → eviction not triggered yet
hit ratio ≈ 95.96% → 4% miss matches "occasional misses"

10.3 Distinguish "More Keys" vs "Larger Keys"

Prefix-count scan (rate-limited SCAN loop) revealed:

3120044 sku:detail
1201111 sku:behavior
400233 sku:price

sku:behavior

only 1.2M keys but consumed most memory. --bigkeys confirmed: Biggest hash "sku:behavior:batch:2024" with 1200000 fields. MEMORY USAGE = 1.32GB (8% of total).

10.4 Root Cause

Code pattern:

key = f"sku:behavior:batch:{year}"
redis.hset(key, mapping={sku_id: json.dumps(behavior)})

Intended "yearly bucket"; became "all SKUs' behaviors for entire year in one Hash". Fields grow linearly with SKU count, never cleaned. Evidence chain: --bigkeys (1.2M fields) + MEMORY USAGE (1.32GB) + code (no TTL) — three independent confirmations.

10.5 Fix: Stop Bleeding, Then Refactor

Stop bleeding : Key no longer written by new code (new version uses daily buckets). UNLINK after full checklist: no writers (code grep + MONITOR sample + OBJECT IDLETIME), backup (batched HSCAN → JSON Lines), low-peak window (02:30), monitoring ready. UNLINK returned 1 (keyspace unlinked), background thread freed memory.

Refactor : New code uses daily keys + 7-day TTL + pipeline HSET + EXPIRE. Added SDK guard: reject HSET with >500 fields.

10.6 Verification

Post-delete: used_memory dropped from 16.9GB to ~15.6GB (watermark 98.8% → 90.9%), stable 30 mins, keyspace_misses growth unchanged, business miss rate flat. If memory rises again → another writer exists; restart triage.

10.7 Rollback Plan

If old version still reads key: restore from JSON Lines via batched HSET (slow but safe for special chars), verify HLEN matches backup line count.

10.8 Postmortem Lessons

Alert threshold too late : 85% left only 1GB buffer. Change to two-tier: 75% warning (ample time), 85% critical.

No periodic large-key patrol : Added weekly --bigkeys -i 0.05 cron, archive to object storage for trend analysis.

HSET lacked size guard : SDK now enforces per-Hash field limit; violations throw exception at source.

10.9 Three Universal Diagnostic Criteria

Three memory fields together : used_memory, used_memory_rss, used_memory_dataset triangle rules out half the hypotheses. Here all three rose in lockstep with stable deltas → pure data growth, no need to chase fragmentation, activedefrag, allocator bugs.

"More keys" vs "Larger keys" first : DBSIZE vs history + prefix count. Normal key count + doubled memory = large key; doubled key count + normal size = growth or missing TTL. Remediation differs entirely.

Confirm writers stopped before delete : Code grep + MONITOR sample + OBJECT IDLETIME all pointing to "no writes". If writers exist, key regrows faster after delete (space freed → faster growth), looks like "delete failed".

Non-technical lesson: First time you raise maxmemory , you should already be hunting root cause. Raising limit bought two weeks but let problem spread from 5 to 3 instances + host swap. Memory issues rarely self-heal; earlier investigation = lower cost.

11. Production Checklist

11.1 Pre-Change Verification

maxmemory set : CONFIG GET maxmemory → non-zero.

Eviction policy matches business : CONFIG GET maxmemory-policy → cache: allkeys-lru / lfu; persistent: noeviction.

Lazyfree evaluated : CONFIG GET lazyfree-* → conscious decision.

Overcommit setting : cat /proc/sys/vm/overcommit_memory → 1 (recommended).

Swap usage : free -m, /proc/<pid>/status | grep VmSwap → zero.

Replication healthy : INFO replication → role, connected_slaves, master_link_status:up.

Backup within RPO : latest RDB/AOF timestamp → within RPO window.

Change window : change ticket → low traffic, avoid bgsave/rewrite.

11.2 Credentials & Permissions

Never pass password via redis-cli -a (leaks in ps /history). Use REDISCLI_AUTH env var. Production: ACL users — read-only for ops (

+@read +@connection +info +memory|usage +scan +client|list -@dangerous

), write-only for change accounts. Test with ACL DRYRUN (7.0+). Password file chmod 600, owned by runtime user, never in repo.

11.3 High-Risk Operations Impact Table

DEL large key : whole instance (sync free) — 100ms–seconds → alternative UNLINK.

FLUSHALL : all data lost — seconds (sync) / long (async) → alternative prefix-batched delete.

KEYS * : whole instance blocked — proportional to key count → alternative SCAN.

CONFIG SET maxmemory lower : immediate mass eviction — second-level jitter → alternative throttle writes first.

DEBUG SLEEP : whole instance blocked — specified duration → alternative disable DEBUG.

MONITOR : CPU spike, log leak — continuous → alternative short + timeout.

SAVE : block until RDB done — tens of seconds → alternative BGSAVE.

Full sync : master fork + bandwidth — minutes → alternative ensure repl-backlog sufficient.

Disable DEBUG / CONFIG via rename-command DEBUG "" / rename-command CONFIG "CONFIG_b8f3a1" in redis.conf (requires restart). Log new names in secret store.

11.4 Observability Baseline

Before go-live, ensure metrics + alerts for: memory watermark, eviction rate, fragmentation ratio, connected/blocked clients, instantaneous ops/sec, hit/miss ratio, RDB/AOF status, replica count & offset lag. Missing any costs an extra hour in next incident.

12. Daily Patrol Script

Converts manual 10-command checks into scheduled report with exit codes (0=OK, 1=WARN, 2=FATAL). Design choices: set -uo pipefail (not -e) so single failures don't abort entire patrol; integer arithmetic via awk (no bc in minimal images); built-in rate limiting (sleep every 5K keys); output archived daily for trend comparison. Cron example: 10 3 * * * /opt/scripts/redis-daily-check.sh ... (escape % in crontab). Run manually 3x before enabling cron; tune thresholds to avoid alert fatigue.

13. Command Quick Reference

Grouped by purpose: memory overview ( INFO memory, MEMORY STATS, MEMORY DOCTOR, MEMORY PURGE), single key ( TYPE, MEMORY USAGE, TTL, OBJECT ENCODING/FREQ/IDLETIME), scan/stats ( DBSIZE, --bigkeys, --memkeys, --hotkeys, INFO keyspace, SLOWLOG GET), config ( CONFIG GET/SET for maxmemory, maxmemory-policy, maxmemory-samples, activedefrag, lazyfree-*, slowlog-*), activedefrag tuning (ignore-bytes, thresholds, cycle-min/max), verification loop (watch frag ratio + ops/sec for 30 mins).

14. Summary

Redis memory issues boil down to three threads:

Why memory grows : Separate three-layer view ( used_memory, used_memory_rss, used_memory_dataset). Divergence → overhead/fragmentation; synchronized rise → data volume. Data volume splits into "more keys" ( DBSIZE + prefix count) vs "larger keys" ( --bigkeys + MEMORY USAGE). All conclusions need ≥2 independent evidences.

What happens at limit : Depends entirely on maxmemory-policy. noeviction → write errors (no data loss); allkeys-* → evicts any key (cache OK, hit rate drops); volatile-* → evicts only TTL keys; if most keys lack TTL, degrades to noeviction — the most misdiagnosed trap. Clarify policy before deleting data.

Large key handling : Criterion = observable latency/bandwidth impact, not element count. Remediation by type: structural split, add TTL, delete debris, fix consumer first, windowed trim. Always UNLINK; collections via HSCAN / SSCAN / LPOP count batched. Every step: backup, canary, metric verification, rollback path.

Operational habits to keep:

Two-tier memory alerts (75%/85%) for response time.

Weekly large-key patrol, archive results for trends — not just reactive scans.

Every large-key deletion logged with impact scope, verification metrics, rollback plan; postmortem conclusions codified into SDK constraints (e.g., field-count limit) or the same problem returns in six months.

Memory governance is a loop: monitor → locate evidence → classify remediation → verify & retrospect → enforce in code. A few cycles and recurring issues vanish.