Skip to content
Redis Production Survival: Big Keys, Hot Keys & the Monitoring That Prevents Outages
Operations 52 min read

Redis Production Survival: Big Keys, Hot Keys & the Monitoring That Prevents Outages

This article dissects three real Redis incidents — an 800MB key deletion that triggered cascading failovers, an 180K QPS hot key that saturated one shard while five idled, and an unmonitored disk that killed AOF rewrites — then delivers a complete playbook for detecting, governing, and observing big keys and hot keys using layered tooling, async deletion, local caching, replica sharding, and a four-tier monitoring stack.

Architecture Pony
Architecture Pony
Architecture Pony

Three Incidents That Reveal the Blind Spots

During a major promotion rehearsal, a 3-million-follower influencer triggered an unfollow-list cleanup. The backend executed DEL on a single large Hash — the main thread blocked for 2.8 seconds . Every request on that shard timed out; Sentinel declared the master subjectively down and triggered failover. The new master replayed the same DEL from the replication stream and blocked another 2.8 seconds, causing a second failover. Four failovers in three minutes exhausted client connection pools. Post-mortem revealed the root cause: no one knew the key was 800 MB .

The second incident was a textbook hot-key case. At 20:00 a flash sale began; operations pointed the entry page to a single SKU. Its inventory key was read 180,000 times per second — but the cluster had six shards, so all traffic hit one shard. That shard hit 100% CPU while the other five sat at 5%. The on-call engineer’s first instinct was to scale out — adding machines did nothing because the sharding rule didn’t change; the hot key still landed on the same shard. The lesson: hot keys are a distribution problem, not a capacity problem . Spreading 180K QPS across six shards yields 30K each — easily handled; concentrating it on one shard kills any node.

The third incident was silent. One afternoon all write APIs errored. Investigation showed AOF rewrite failed — disk full . The disk had been filled by a runaway logging component, and no one monitored the Redis host’s disk usage . Two weeks earlier INFO had already shown aof_last_bgrewrite_status flip to aof_rewrite_concurrent_bgsave_error — nobody looked . The common thread across all three: Redis didn’t break; the blindness did — big keys undiscovered, hot spots unpredicted, metrics unsubscribed.

Technical Analysis: Definition, Detection, Governance

1.1 Big Keys: How Big Is Big, Why Dangerous

Actionable thresholds (industry standard, tighten as needed):

String value > 10 KB

Collection types (Hash/List/Set/ZSet): element count > 5,000 or total serialized size > 1 MB

Danger stems from Redis’s single-threaded command processing — every operation on a big key (read, write, expiry deletion, migration) takes time proportional to its size. An 800 MB DEL blocks the main thread for seconds, queuing all requests.

Four harm chains:

Command blocking — DEL / HGETALL / SMEMBERS are all O(N).

Replication amplification — writing a big key once sends 800 MB down the replication link; the replica replays it and blocks again (the four failovers in incident one).

Cluster data skew — the shard holding the big key monopolizes memory and traffic.

Memory fragmentation & COW amplification — updating a big key triggers fragmentation compaction; during persistence fork it resides in many pages, multiplying copy-on-write overhead.

1.2 Hot Keys: The Solo Act on One Shard

Detection thresholds:

Single-key QPS > 10% of a shard’s designed capacity (e.g., shard rated 50K QPS → watch keys > 5K).

Cluster traffic Gini imbalance — top-1 key traffic > 20× the mean.

Harm chain: single shard CPU saturates → all keys on that shard (not just the hot one) see latency spikes → client timeouts & retries → retry traffic further heats the shard → avalanche.

It’s a distribution problem, not capacity. Adding machines or memory doesn’t help because the sharding rule is unchanged — the hot key always lands on the same shard. Only two fundamental fixes:

Block requests before Redis (local cache).

Turn one key into many (replica sharding).

1.3 Detecting Big Keys: Four-Layer Probe Net

Ordered by cost (low → high):

redis-cli --bigkeys — built-in type-level scan, production-safe (sampling, non-blocking), but only reports the largest key per type, not full distribution.

RDB offline analysis — parse RDB with rdb-tools / redis-rdb-cli; full, precise, zero online impact, but a point-in-time snapshot.

MEMORY USAGE command — precise sizing of suspicious keys (includes internal overhead), paired with known-key inventory for targeted checkups.

Client-side instrumentation — measure serialized value size at the serialization layer, alert on threshold breach. This is the only way to intercept at write time — the best governance moment is when the big key is born, not when it’s 800 MB.

Routine: layers 1–2 run weekly with reports; layer 4 lives in production.

1.4 Detecting Hot Keys: From Server to Client

Four parallel paths:

redis-cli --hotkeys — relies on LFU access counts; works under allkeys-lfu, inaccurate under LRU.

Client/proxy-layer stats — Lettuce/Jedis command listeners, proxy (Codis, custom) key-level counters. Most accurate position because all traffic passes here .

MONITOR short sampling — high-risk to run continuously; open for a few seconds during emergencies.

Business-side prediction — ops provides promo SKU list before big sales; proactively warm local caches and shard these keys. This is the only pre-event lever. Incident two’s core finding: hot-key detection cannot rely solely on post-hoc discovery; business events (promos, viral items, trending topics) must trigger hot-key playbooks .

1.5 Governing Big Keys: Split, Compress, Delete

Split — shard large collections by business key: fanout Hash split by user_id % 100 into user:follow:{0..99}, each bucket a few thousand elements; use Cluster hash tags to keep same-rule keys from scattering across too many shards.

Compress — oversized String values: compress first (e.g., protobuf + zstd, typically 5–10×), or move “cold fields inside the big object” back to DB and cache only hot fields.

Delete — must use UNLINK (async: main thread only unlinks dict entry, actual memory freed by background BIO threads), and enable all four lazyfree switches ( lazyfree-lazy-expire/evict/server-del/replica-lazy-flush). Stock cleanup: SCAN + UNLINK in small batches, never naked DEL .

Common prerequisite: “quantify first.” MEMORY USAGE confirms size, TYPE confirms structure, then choose tactic — for an 800 MB Hash, HSCAN batched reads + deletes is safer than a single UNLINK.

1.6 Governing Hot Keys: Local Cache + Replica Sharding

Local cache — in-process Caffeine, hot keys read locally, TTL seconds (1–5 s), consistency via short TTL natural expiry or invalidation broadcast. It cuts 180K QPS to “1–2 origin requests per app instance per second” — highest ROI. Discipline from consistency chapter: local cache only for extremely hot keys that tolerate second-level staleness .

Replica sharding — turn stock:1001 into stock:1001:{0..4} five replicas, reads randomly scattered. Writes must preserve total consistency — either split inventory into five equal buckets (true per-shard decrement) or write single key, read multiple replicas.

Standard combo: local cache blocks reads, replica sharding handles read skew when writes can’t be split, rate-limiting as final safety net.

Often overlooked option: move the hot key’s data back to app memory or direct DB read — it may be hot precisely because caching it was never worthwhile.

1.7 Monitoring System: Four-Layer Metric Net

Consolidating monitoring points from the series into four layers:

Resource layer — memory watermark ( used_memory/maxmemory, RSS), CPU, disk (AOF disk watermark, incident three’s blind spot), network bandwidth (replication storm’s first signal).

Performance layer — latency (latency monitor + P99), slowlog ( slowlog-log-slower-than recommended 1 ms; empty slowlog ≠ no problem, threshold may be wrong), hit rate & evicted_keys.

Health layer — replication offset lag ( master_repl_offset diff), Sentinel events, Cluster PFAIL / FAIL & slot coverage.

Business layer — scheduled big-key scan reports, hot-key leaderboards, per-instance key-count & memory growth curves.

Alert discipline three rules: every alert has an owner, weekly alert-accuracy review, unread metrics = unconfigured. Monitoring’s value isn’t pretty dashboards — it’s someone receiving that message 30 minutes before the outage .

Core Workflows: Delete, Warm, Split, Troubleshoot

2.1 Correct Sequence for Big-Key Deletion

Standard flow for an 800 MB fanout Hash:

T1 MEMORY USAGE confirm size & type.

T2 Start during business low-traffic window.

T3 If clearing data only: HSCAN cursor batched (1,000 elements/batch) → HDEL per batch → after all batches, DEL empty shell key — every batch is millisecond-scale, main thread unaffected.

T4 If deleting key entirely and no concurrent reads: UNLINK async delete; main thread only unlinks dict reference O(1), real memory release by BIO background threads (controlled by lazyfree-lazy-user-del & UNLINK queue).

T5 Watch replica replay & memory curve, confirm no fragmentation spike.

Contrast with incident one’s wrong path: naked DEL 800 MB → main thread blocks 2.8 s → full-shard timeouts → Sentinel misjudges → cascade failovers. One command choice = one site-wide outage.

2.2 Online Migration for Big-Key Split

Split a 3M-fanout Hash into 100 buckets without stopping writes — seven steps:

Create new bucket structures user:follow:{0..99}.

Enable dual-write — write path writes both old Hash and new buckets (new logic failure doesn’t block old path). HSCAN full backfill of historical data into buckets (batched, rate-limited, avoid peak).

Data validation — sample compare old Hash vs new buckets’ element counts & content (align sum of HLEN s).

Switch read path to new buckets (canary by app instance rolling).

Observe one full business cycle with no rollback need. UNLINK old Hash.

Safety net throughout: old key retained until step 7; any step can roll back to reading old key — online migration’s safety comes entirely from “old data still exists.”

2.3 Hot-Key Playbook Standard Action Sequence

24 hours before big promo, execute in order:

Get ops’ promo SKU list, annotate estimated QPS.

For keys > 5K QPS, decide per key: read-hot → add to local-cache whitelist & prewarm; write-hot → confirm inventory decrement scheme (shard inventory or DB conditional update).

Prewarm local cache — batch GET hot data before kickoff to avoid thundering-herd miss storms at launch.

Confirm rate-limit thresholds sit “before origin” (chapter seven discipline).

On-call knows where hot-key dashboard lives, which keys trigger which playbook.

Post-event: compare actual hot-key ranking vs prediction, calibrate estimation model. Had incident two followed this, the 180K QPS inventory key would have been blocked by local cache before launch.

2.4 Latency Spike Troubleshooting Path

P99 jumps from 2 ms to 50 ms — standard investigation order (by hit rate):

Latency monitor event list — directly records “what slow event happened when.”

Slowlog — any 10 ms+ commands? O(N) large-collection commands are suspect #1.

Exclude persistence — AOF everysec fsync stalls ( aof_delayed_fsync counter), RDB fork time ( latest_fork_usec), COW page faults during rewrite.

Exclude big keys — recent expiry deletions (lazyfree enabled?), large-collection HGETALL.

Exclude OS layer — host CPU contention, memory swap, network retransmits.

Exclude hot keys — single-shard CPU spike?

Value of the sequence: check Redis’s own records (monitor/slowlog) first, then drill down — avoids jumping straight to network blame.

Core Source Code: Five Key Implementations

3.1 Big-Key Bucket Split: HSCAN Progressive Backfill

// 3M-fanout Hash split into 100 buckets
String cur = "0";
do {
  ScanResult<Map.Entry<String,String>> r
    = redis.hscan("follow:big", cur,
       new ScanParams().count(1000));
  cur = r.getCursor();
  Pipeline p = redis.pipelined();
  for (var e : r.getResult()) {
    int b = Math.floorMod(
      e.getKey().hashCode(), 100);
    p.hset("follow:" + b,
      e.getKey(), e.getValue());
  }
  p.sync();  // each batch millisecond-level
} while (!"0".equals(cur));
// Validation: sum of 100 buckets' HLEN == old HLEN

3.2 Safe Delete & Lazyfree Config

# redis.conf four switches all on
lazyfree-lazy-expire yes
lazyfree-lazy-eviction yes
lazyfree-lazy-server-del yes
replica-lazy-flush yes

# Stock cleanup: SCAN+UNLINK progressive
SCAN 0 MATCH follow:* COUNT 100
# UNLINK each (async free)
UNLINK follow:big
# Main thread only unlinks ref O(1);
# Large object freed by BIO bg thread
# NEVER: DEL follow:big naked

3.3 Hot-Key Local Cache: Caffeine + Version Invalidation

// Only for extremely hot keys tolerating second-level staleness
Cache<String,String> hot =
  Caffeine.newBuilder()
  .maximumSize(5_000)
  .expireAfterWrite(2, SECONDS)
  .recordStats()
  .build();

String get(String key) {
  String v = hot.getIfPresent(key);
  if (v != null) return v;
  v = redis.get(key);  // origin
  if (v != null) hot.put(key, v);
  return v;
}
// Second-level TTL natural expiry; hard invalidation via
// Redis pub-sub broadcast "delete local"
// Hit rate into monitoring, <80% triggers review

3.4 Hot-Key Replica Sharding: Read Scatter & Inventory Split

// Read scatter: N replicas random read
int n = ThreadLocalRandom.current()
  .nextInt(5);
String v = redis.get(
  "stock:1001:" + n);

// Write split: inventory cut into five, each 1/5
int b = orderId.hashCode() % 5;
Long ok = redis.eval(
  "if redis.call('GET',KEYS[1])"
  + ">&= tonumber(ARGV[1]) then "
  + "return redis.call('DECRBY',"
  + "KEYS[1],ARGV[1]) end return 0",
  1, "stock:1001:" + b, qty);
// Single bucket exhausted → cross-bucket borrow or fail-degrade
// Fallback: DB conditional update for final correctness

3.5 Routine Checkup Script: bigkeys + INFO Collection

import subprocess, re
# 1) Big-key scan (sampling, online-safe)
out = subprocess.run(
  ["redis-cli","--bigkeys","-i","0.1"],
  capture_output=True, text=True).stdout
print(out[-800:])  # largest key per type

# 2) INFO key metrics collection
info = subprocess.run(
  ["redis-cli","INFO","memory"],
  capture_output=True, text=True).stdout
used = re.search(r"used_memory:(\d+)",
  info).group(1)
maxm = re.search(r"maxmemory:(\d+)",
  info).group(1)
ratio = int(used) / max(int(maxm), 1)
if ratio > 0.85:
  alert("memory", ratio)  # alert

The script’s value isn’t one-off debugging — it’s routine + reporting : run weekly, push “per-type top-10 big keys + memory watermark + key-count MoM” to the team channel. Big-key report MoM trends beat absolute values — a key growing from 2 MB to 200 MB would have alerted on the curve long before the DEL exploded.

Advanced Topics: Six Accounts You Must Pre-Calculate

4.1 Preventing Big Keys: Three Gates at Write Path

Governance = split; prevention = write-path constraints. Three gates by cost:

Client size gate — unified wrapper serializes, measures; >10 KB logs warning, >100 KB rejects write and demands split design.

Collection count gate — before Hash/List/Set write, check HLEN / LLEN; >5,000 prompts bucketing.

Structure review gate — “why is this in Redis?” review: hot data or misused as storage? Naturally growing lists (fans, order streams) must be designed with bucket model from day one, not split after hitting 3M . Incident one’s 800 MB Hash would never have existed as a single key if design review asked “estimated fan count?”

4.2 Hot-Key Detection Level Selection

How to combine the four probes depends on phase:

Daily — proxy/client stats resident (most accurate, zero Redis intrusion) + weekly --hotkeys report.

Pre-promo — business whitelist prediction + local cache prewarm; this layer essentially “replaces technical discovery with ops intelligence.”

During incident — MONITOR second-level sampling + single-shard CPU comparison, fast hot-key lock.

Post-mortem — feed actual hot keys back into business prediction whitelist.

Often missed fact: client-side detection sees “which key read by which service” — only it can link hot key to business owner ; Redis server only sees key names, not owners. Governance ultimately requires business-side code changes, so “find the owner” must come from client instrumentation.

4.3 Cluster Skew: Slot & Key Distribution Accounts

Two skew types, different mechanisms, different fixes:

Memory skew — big key or hash-tag clustering makes one shard’s memory uniquely high; check CLUSTER NODES slot distribution & per-shard key counts, split keys, scatter tags.

Traffic skew — hot key saturates one shard; local cache / replica sharding.

Planning-level account: slot allocation must reserve imbalance buffer — 16,384 slots manually allocated; if fast-growing business line shares shards, it will crowd out neighbors. Allocate slot segments by business line (and leave empty slots as buffer) — the single most valuable action during Cluster planning.

Hash tag’s double edge: {user1001}:profile + {user1001}:orders same slot enables multi-key ops, but also locks that key pair’s traffic forever on one shard — tag cardinality (per chapter five criteria) = concurrency ceiling .

4.4 Slowlog Correct Usage: Threshold, Sampling, Misreads

Engineering calibrations for three params:

slowlog-log-slower-than = 1,000 µs (1 ms) — default 10 ms too wide; by the time it alerts, users already feel it.

slowlog-max-len = 1,024 — default 128 insufficient during jitter storms.

It’s an in-memory queue, cleared on restart; must periodically SLOWLOG GET to archive.

Two frequent misreads: empty slowlog ≠ no problem — threshold wrong or stall from non-command paths (fsync, fork, expiry storm — these don’t enter slowlog); KEYS in slowlog ≠ immediate global ban — KEYS on 10-key test DB is innocent; criterion is keyspace size.

Treat slowlog as “command-layer thermometer,” latency monitor as “system-layer medical record” — latter captures fork time, fsync stalls invisible to commands; both complementary, neither optional.

4.5 Capacity Planning: Watermark & Growth Forecast

Four accounts:

Memory watermark redline 70% — breach triggers scale review, don’t wait for evicted_keys (chapter six discipline).

Growth forecast — weekly key-count & memory MoM; monthly growth > 20% → instance hits wall within six months, schedule early.

Peak headroom — promo/month-end batch peak multiples must be rehearsed; 60% watermark at 3× peak = 180%, guaranteed death.

Fork headroom — persistence-enabled instances: COW worst-case doubles memory (chapter three account), maxmemory must not exceed 80% of physical RAM.

Scale levers by cost: clean big keys → tune TTL → split instances → add shards — do free ones first, then talk hardware. Incident three’s full disk = capacity account missed “non-Redis processes”: capacity planning must cover the whole host, not just Redis.

4.6 Production Checklist: One Page Guards the Whole Series

Condense entire Redis series discipline into one quarterly巡检 checklist:

Persistence — RDB/AOF status & recent rewrite results, fsync latency counters, fork time, disk watermark.

HA — master-replica offset < 5%, Sentinel quorum intact, Cluster slots fully covered & PFAIL zero.

Memory — watermark < 70%, evicted_keys</strong> no growth, no unexpected <code>TTL -1 keys.

Performance — slowlog (1 ms threshold) no new O(N) commands, latency monitor no new events, P99 MoM stable.

Structure — this week’s big-key report no new entries, hot-key leaderboard no surprise faces.

Security — requirepass /ACL complete, dangerous commands renamed effective, lock instance & cache instance physically isolated & lock instance has no eviction policy.

Checklist’s purpose: turn “experience” into “actions” — every line maps to a real incident from the series.

Learning Path: From Firefighting to Fire Prevention

Week 1: Full checkup on existing instances — --bigkeys + RDB offline analysis + MEMORY USAGE precision on top keys → first big-key report; set slowlog threshold to 1 ms, observe one week.

Weeks 2–3: Build four-layer monitoring — INFO into TSDB, latency monitor on, slowlog periodic archive, weekly checkup script live, wire alert ownership flow.

Weeks 4–6: Clear stock — “split/compress/delete” on reported big keys (first UNLINK risky keys, then schedule bucket migrations), put top hot keys behind local cache.

Deep dive: Read two things — redis-rdb-tools parsing logic (understand RDB format & memory estimation) and Redis source bio.c background task queue (understand UNLINK /lazyfree async path) — close the “what happens after command returns” knowledge gap.

High-Frequency Interview Questions (with Summary Answers)

20 questions covering all chapter points; answers lead with conclusion then reasoning, ready for interview phrasing.

Q1: What is a big key? Harms? Thresholds: String >10 KB; collection >5K elements or >1 MB. Four harms: ① command blocking (single-threaded, O(N) blocks all), ② replication amplification (write once, replicate 800 MB, replica replays & blocks again), ③ cluster skew, ④ fragmentation & COW amplification. All root-caused in “single-threaded + time proportional to size.”

Q2: How to detect big keys? Pros/cons? Four layers: ① --bigkeys sampling, safe but only max per type; ② RDB offline (rdb-tools) full, precise, zero impact but snapshot; ③ MEMORY USAGE precise sizing (incl. overhead); ④ client serialization instrumentation — only way to intercept at write time. Combo: 1–2 weekly, 4 resident. Best governance moment = birth.

Q3: Why UNLINK not DEL for big keys? DEL sync frees — 800 MB Hash blocks main thread seconds, full-shard timeouts, Sentinel misjudge, cascade failover. UNLINK main thread only unlinks dict O(1), real free by BIO bg threads. Enable four lazyfree switches (expire/eviction/server-del/replica-flush) so expiry, eviction, implicit delete, full-sync flush all go async. Stock cleanup: SCAN + UNLINK progressive, never naked DEL.

Q4: How to govern big keys? Three mantras: Split — bucket by business key (fanout Hash → 100 buckets), HSCAN progressive backfill + dual-write + validate + canary read switch + UNLINK old key (7-step online migration). Compress — value compress (protobuf+zstd 5–10×) or cold fields back to DB. Delete — UNLINK /lazyfree async, SCAN batched. Prerequisite: quantify first ( MEMORY USAGE, TYPE). Prevention: three write-path gates (size, count, structure review). Core idea: best governance moment = birth.

Q5: What is a hot key? Why scaling out fails? Threshold: single-key QPS >10% shard capacity, or top-1 traffic >20× mean. It’s a distribution problem, not capacity — sharding rule unchanged, hot key always lands same shard; scaling only dilutes other keys. Harm chain: shard CPU full → all keys on that shard latency spikes → timeouts/retries → retry traffic further heats shard → avalanche. Two fundamental fixes: local cache blocks requests outside Redis, or replica sharding turns one key into many.

Q6: How to detect hot keys? Four paths: ① --hotkeys (LFU counts, inaccurate under LRU); ② client/proxy key-level stats (most accurate, only one linking hot key to business owner); ③ MONITOR short sampling (emergency only); ④ business prediction — promo SKU list triggers playbook, only pre-event lever. Phase-based combo: daily ②+④, pre-promo ④+prewarm, incident ①+③.

Q7: How to govern hot keys? Three strikes: ① Local cache (Caffeine, second-level TTL + invalidation broadcast) — 180K QPS → 1–2 origin req/app/sec, highest ROI, only for extremely hot + second-stale-tolerant. ② Replica sharding — stock:1001:{0..4} random read scatter; write-hot split inventory five ways, cross-bucket borrow on exhaustion. ③ Rate-limit at origin. Combo: local blocks reads, replica handles read skew when write can’t split, rate-limit safety net. Reflection: key may be hot because it never belonged in Redis.

Q8: Cluster skew types? Two: Memory skew — big key or hash-tag cluster → one shard memory high; check slot dist & key counts, split keys, scatter tags. Traffic skew — hot key saturates shard; local cache / replica sharding. Planning action: allocate slot segments by business line + empty slot buffer. Hash tag double edge: same-slot enables multi-key ops but locks traffic to one shard — tag cardinality = concurrency ceiling.

Q9: Latency spike standard troubleshooting path? By hit rate: ① latency monitor slow events; ② slowlog O(N) large-collection commands; ③ exclude persistence — fsync stalls ( aof_delayed_fsync), fork time ( latest_fork_usec), rewrite COW; ④ exclude big keys — expiry deletes, HGETALL; ⑤ OS layer — CPU contention, swap, net retransmit; ⑥ hot key — single-shard CPU spike. Discipline: Redis’s own records first, then drill down — don’t start blaming network.

Q10: Slowlog config? Misreads? Calibrations: slowlog-log-slower-than 1,000 µs (default 10 ms too wide), slowlog-max-len 1,024 (default 128 too small), in-memory queue cleared on restart → periodic SLOWLOG GET archive. Misreads: empty ≠ no problem (threshold wrong or stall from fsync/fork/expiry storm — non-command paths); KEYS present ≠ instant ban (criterion = keyspace size). Slowlog = command thermometer, latency monitor = system medical record — complementary.

Q11: Big-key prevention measures? Three write-path gates: ① Client size gate — unified wrapper serializes & measures, >10 KB warn, >100 KB reject & demand split. ② Collection count gate — HLEN / LLEN >5K prompt bucket. ③ Structure review gate — “why Redis?” review: naturally growing data (fans, streams) must use bucket model at design, not split at 3M. Core: best governance moment = birth.

Q12: Production Redis metrics to watch? Four layers: Resource — memory watermark, CPU, disk (incl. AOF disk), net BW. Performance — latency monitor+P99, slowlog, hit rate, evicted_keys. Health — replication offset lag, Sentinel events, Cluster PFAIL / FAIL & slot coverage. Business — weekly big-key report, hot-key leaderboard, key-count & memory growth curves. Alert discipline: every alert owned, weekly accuracy review, unread = unconfigured. Monitoring value = that message 30 min before outage.

Q13: Capacity planning accounts? Four: ① Watermark redline 70%, breach → scale review, don’t wait for evicted_keys. ② Growth forecast — weekly MoM, monthly >20% → hit wall in 6 months, schedule early. ③ Peak headroom — promo 3× peak rehearsal; 60% watermark at 3× = 180%, dead. ④ Fork headroom — persistence COW worst-case 2×, maxmemory ≤80% physical. Scale levers by cost: clean big keys → tune TTL → split instances → add shards. Must account whole host, not just Redis (disk full by other process lesson).

Q14: Why SCAN safe, KEYS dangerous? KEYS full traversal single return — large keyspace blocks main thread seconds. SCAN cursor-based batches ( COUNT controls batch), each batch millisecond, main thread unaffected, mid-traversal keyspace changes won’t miss (may duplicate, app-layer dedup). Same for large collections: HSCAN / SSCAN / ZSCAN replace HGETALL / SMEMBERS; big-key cleanup: SCAN + UNLINK progressive. All “full” actions → “cursor + rate-limit.”

Q15: Online big-key migration safety? Seven steps: create buckets → dual-write (new logic failure doesn’t block old) → HSCAN batched rate-limited backfill → sample validate (bucket HLEN sum vs old HLEN) → canary read switch → observe full cycle → UNLINK old key. Safety source: old key kept until step 7, any step can rollback to old read; write path dual-write all the way — no incremental loss. No migrations before promos.

Q16: Lazyfree configs & scope? Four switches + one command: lazyfree-lazy-expire (expiry async), lazyfree-lazy-eviction (eviction async), lazyfree-lazy-server-del (implicit delete e.g. RENAME overwrite async), replica-lazy-flush (full-sync pre-flush async); explicit UNLINK anytime. Shared mechanism: main thread only dict unlink O(1), large object free by BIO bg threads. IO threads (Redis 6+) can further parallelize free.

Q17: Pre-promo Redis playbook checklist? ① Ops promo SKU list → annotate est. QPS; ② >5K QPS keys per-key decision: read-hot → local cache whitelist + prewarm; write-hot → confirm inventory decrement scheme; ③ prewarm local cache before launch to avoid thundering miss; ④ rate-limit thresholds at “before origin”; ⑤ on-call knows hot-key dashboard & playbook mapping; ⑥ capacity rehearsal 3× peak; ⑦ degrade drill (chapter seven). Post: compare actual hot-key ranking vs prediction, calibrate model. Playbook value = turning defense validation into verifiable actions.

Q18: INFO metrics worth staring at? By signal strength: used_memory/maxmemory watermark + mem_fragmentation_ratio (>1.5 fragmentation anomaly); evicted_keys growth (chapter six: >0/min = alert); master_repl_offset vs replica diff (replication health); aof_delayed_fsync & latest_fork_usec (persistence backfire); rejected_connections (connection layer); instantaneous_ops & per-sec per-shard compare (skew). INFO is free, updates every second — if someone collects & watches.

Q19: Must-have prod config hardening? ① Auth & ACL: requirepass /ACL, no public bind; ② Dangerous commands: KEYS / FLUSHALL / CONFIG renamed to empty or ACL-restricted; ③ Lazyfree four switches on; ④ Slowlog 1 ms/1,024 + latency monitor on; ⑤ Lock instance & cache instance physically isolated, lock instance forbids maxmemory eviction (chapter nine); ⑥ Replica read-only, protected mode; ⑦ maxmemory leaves COW headroom per persistence type. Each maps to a real incident class, not style preference.

Q20: Big-key vs hot-key governance priority? By “speed to cause outage”: 1) Stop bleeding — naked DEL on slowlog big keys (one op can nuke site), enable UNLINK + four switches, lowest cost highest ROI. 2) Cure residents — schedule bucket migrations for report’s top big keys (online 7-step). 3) Block hotspots — top hot keys behind local cache + replica sharding. 4) Harden — three write gates + weekly checkup + promo playbook, so problems stop being born instead of continuous firefighting. Principle: stop bleeding first, then debride, finally change lifestyle.

One-Page Cheat Sheet

Big-key three thresholds: String >10 KB, collection >5K elements or >1 MB, single-threaded era every op time ∝ size.

Delete one commandment: UNLINK + lazyfree four switches, naked DEL on 800 MB key = one site-wide outage.

Detect four-layer net: bigkeys routine, RDB offline full, MEMORY USAGE precision, client instrumentation intercept at birth.

Govern three mantras: Split (bucket + 7-step online migration), Compress (zstd 5–10×), Delete ( SCAN + UNLINK progressive).

Hot key in one line: Distribution problem not capacity problem, scaling useless — local cache blocks reads, replica sharding scatters, rate-limit safety net.

Monitor four layers: Resource (watermark/CPU/disk), Performance (slowlog 1 ms + latency monitor), Health (replication offset/slot coverage), Business (weekly big-key report/hot-key leaderboard).

Troubleshoot one path: latency monitor → slowlog → persistence (fsync/fork) → big key → OS layer → hot key.

Closing line: 90% of Redis problems aren’t Redis broken — it’s that nobody saw. Monitoring’s value = that message 30 minutes before the incident.

Summary & Series Finale

This chapter completes the last three pieces of the production puzzle: big-key definition, four-layer detection & “split/compress/delete” governance; hot-key distribution nature & “local cache + replica sharding + rate-limit” combo; four-layer monitoring & latency spike troubleshooting path. One sentence to review the whole series: chapters 1–2 explain why it’s fast (single-thread + efficient structures), 3–6 explain how data lives (persistence, replication, cluster, eviction), 7–8 explain how cache defends (three defenses & consistency), 9 explains how locks work (efficiency tool & correctness boundaries), this chapter explains how ops guards (detect, govern, monitor). Ten chapters form one complete thesis: Using Redis well isn’t memorizing commands — it’s understanding the cost behind every design trade-off and turning cost management into制度. Thanks for following along; next we open a new series — Nginx & Gateway Layer Performance in Practice , from worker model to rate-limiting & circuit-breaking.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

monitoringRediscapacity planningbig keysproduction operationshot keyscluster shardinglatency troubleshooting
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.