Redis Production Governance: Big Keys, Hot Keys, Memory & Scaling

Three Production Incidents

The article opens with three real incidents that illustrate Redis's two fundamental limits in a sharded architecture: how long a single command can block the main thread, and how much load a single shard can handle.

1. HGETALL Blocks an Entire Shard

An order service stored user orders in a Hash with key order:list:{userId}. A head user accumulated 800,000 fields. A data export job ran HGETALL order:list:123, which occupied the main thread for 2.8 seconds. Because the key's slot resided on one shard, all other requests to that shard queued up, generating 3,000+ Jedis timeout logs and causing upstream cascade failure.

The key insight: the danger of a big key is not its memory footprint but the fact that operating on it blocks the entire shard.

2. Hot Key Drives Single-Shard CPU to 100%

During a Double 11 pre-sale, a hot product detail key reached 320,000 QPS, pushing instantaneous_ops_per_sec to 400,000 on its shard and saturating CPU. The other 1/8 of business on the same shard suffered. Adding replicas did not help because Redis Cluster reads default to the primary; clients must explicitly send READONLY to read from replicas.

3. Memory Fragmentation + Wrong Eviction Policy

A service showed used_memory 14 GB, used_memory_rss 22 GB, mem_fragmentation_ratio 1.57 on a 32 GB machine, triggering swap. Meanwhile maxmemory-policy was set to noeviction, so once memory filled, all writes returned OOM command not allowed when used memory > 'maxmemory', breaking the logging pipeline.

Root cause: no routine monitoring; teams only react after incidents.

Governance Loop: Detect → Analyze → Optimize → Protect → Inspect

Ad-hoc --bigkeys scans or short MONITOR sessions treat symptoms, not causes. A sustainable loop requires five continuous stages:

Detect : Automated discovery of anomalous keys and instances via SCAN sampling, RDB offline analysis, client instrumentation, and INFO collection. Output: big-key leaderboards, hot-key leaderboards, metric trends.

Analyze : Root-cause attribution by aggregating across type, business domain, and DB. Map keys to application owners — otherwise a list like order:1234 is unactionable.

Optimize : Eliminate hazards via splitting, compression, UNLINK, local caching, sharding. Each change requires a ticket, canary validation, then full rollout.

Protect : Prevent recurrence with pre-commit gates, SDK interception, capacity watermark alerts.

Inspect : Continuous safety net via scheduled jobs, daily reports, alert linkage.

Detection is most often neglected; it must be automated and continuous.

Big Key Detection: Trade-offs Among Approaches

Threshold Reference

String: value > 1 MB (some teams tighten to 100 KB).

Hash: field count > 5,000 or MEMORY USAGE > 1 MB.

List/Set/ZSet: element count > 5,000 or memory > 1 MB.

For String, either length threshold or byte threshold — using both indicates unclear thinking.

SCAN Sampling (Online, Most Common)

KEYS *

is a production accident generator; must use cursor-based SCAN. Full scan of 50M keys can take minutes and consume CPU/memory. Practice: sample up to N keys per run (e.g., 200,000), use MATCH prefix to narrow scope, run MEMORY USAGE only on suspects, scan each shard independently.

Pitfall : Spring Data Redis's RedisClusterConnection.keyCommands().scan(options) iterates all nodes internally; wrapping another per-node loop scans the cluster N times. Correct: use the node-specific overload scan(node, options). Also MEMORY USAGE is not exposed in ClusterServerCommands; drop to Lettuce native connection.

@Slf4j
@Component
public class BigKeyScanner {
    private static final int SAMPLE_LIMIT = 200_000;
    private static final int BATCH = 1000;
    private final RedisClusterClient clusterClient;

    public BigKeyScanner(LettuceConnectionFactory factory) {
        this.clusterClient = (RedisClusterClient) factory.getNativeClient();
    }

    /** Cluster: scan per node, return node -> big key list */
    public Map<String, List<BigKeyItem>> scanCluster(int thresholdBytes) {
        Map<String, List<BigKeyItem>> result = new LinkedHashMap<>();
        try (StatefulRedisClusterConnection<byte[], byte[]> conn = clusterClient.connect()) {
            for (RedisClusterNode node : conn.getPartitions()) {
                if (node.is(RedisClusterNode.NodeFlag.MASTER)) {
                    String nodeId = node.getUri().getHost() + ":" + node.getUri().getPort();
                    result.put(nodeId, scanNode(conn, node, thresholdBytes));
                }
            }
            return result;
        }
    }

    private List<BigKeyItem> scanNode(StatefulRedisClusterConnection<byte[], byte[]> conn,
                                        RedisClusterNode node, int thresholdBytes) {
        List<BigKeyItem> list = new ArrayList<>();
        StatefulRedisConnection<byte[], byte[]> nodeConn = conn.getConnection(node.getNodeId());
        RedisCommands<byte[], byte[]> cmd = nodeConn.sync();

        int scanned = 0;
        KeyScanCursor<byte[]> cursor = cmd.scan(ScanArgs.Builder.matches("*").limit(BATCH));
        while (scanned < SAMPLE_LIMIT) {
            for (byte[] key : cursor.getKeys()) {
                scanned++;
                // Level-1 filter: MEMORY USAGE, samples=5 to reduce overhead
                Long bytes = cmd.memoryUsage(key, 5);
                if (bytes == null || bytes < thresholdBytes) continue;
                // Level-2 analysis: get element count by type
                String type = cmd.type(key);
                Long len = elementCount(cmd, key, type);
                list.add(new BigKeyItem(
                    new String(key, StandardCharsets.UTF_8),
                    type == null ? "unknown" : type,
                    bytes, len));
            }
            if (cursor.isFinished()) break;
            cursor = cmd.scan(cursor, ScanArgs.Builder.matches("*").limit(BATCH));
        }
        list.sort(Comparator.comparingLong(BigKeyItem::getBytes).reversed());
        return list;
    }

    private Long elementCount(RedisCommands<byte[], byte[]> cmd, byte[] key, String type) {
        if (type == null) return null;
        switch (type) {
            case "hash":  return cmd.hlen(key);
            case "list":  return cmd.llen(key);
            case "set":   return cmd.scard(key);
            case "zset":  return cmd.zcard(key);
            case "string": return cmd.strlen(key);
            default:       return null;
        }
    }
}

This uses Lettuce native cluster API; RedisClusterClient can be obtained from LettuceConnectionFactory.getNativeClient() or a dedicated bean. MEMORY USAGE samples Hash/List/Set/ZSet (smaller samples = faster, higher error; 5 is sufficient for governance). String ignores the samples parameter.

RDB Offline Analysis

Online sampling can miss keys. For 100% coverage, run BGSAVE during low traffic, pull the RDB, and parse with rdbtools or a custom parser:

# Generate memory report (sorted by key type and size)
rdb -c memory dump.rdb -f memory.csv
rdb -c memory dump.rdb --bytes 10240 -f big_keys.csv

# Aggregate by key prefix to quickly locate business owners
awk -F, 'NR>1 {split($3,a,":"); agg[a[1]] += $4} END {for (k in agg) print k, agg[k]}' 
  memory.csv | sort -k2 -nr | head -20

Accurate, non-intrusive, and can pinpoint memory consumption per prefix. Cost: RDB disk I/O plus parse time; best run on a dedicated read-only replica or offline machine.

Prefix Aggregation

Raw big-key lists (e.g., order:1234, order:1235) are unactionable without prefix aggregation. Normalize prefixes and compute Top N:

public List<PrefixStat> aggregateByPrefix(List<BigKeyItem> items) {
    Map<String, PrefixStat> map = new HashMap<>();
    for (BigKeyItem item : items) {
        String prefix = extractPrefix(item.getKey()); // take "biz:module" two levels
        map.computeIfAbsent(prefix, k -> new PrefixStat(k))
             .add(item.getBytes());
    }
    return map.values().stream()
        .sorted(Comparator.comparingLong(PrefixStat::getTotalBytes).reversed())
        .limit(20)
        .collect(Collectors.toList());
}

private String extractPrefix(String key) {
    String[] parts = key.split(":");
    return parts.length >= 2 ? parts[0] + ":" + parts[1] : parts[0];
}

Hot Key Detection: Client Instrumentation Is Primary

MONITOR

and --hotkeys are either too heavy (MONITOR echoes every command; client collapses at 100k QPS) or too coarse ( --hotkeys relies on LFU, requires maxmemory-policy=allkeys-lfu, counters decay over time). Production-grade approach: client-side instrumentation as main, proxy-layer stats if available.

Client-Side Sliding Window

Wrap the Redis SDK to count per-key access in a second-level sliding window, report Top N every 10 seconds:

public class HotKeyDetector {
    private static final int WINDOW_SECONDS = 60;
    /** One sliding window per key */
    private final ConcurrentHashMap<String, SlidingWindow> windows = new ConcurrentHashMap<>();

    public void record(String key) {
        if (windows.size() > 100_000) { // prevent memory bloat, degrade silently
            return;
        }
        windows.computeIfAbsent(key, k -> new SlidingWindow(WINDOW_SECONDS)).incr();
    }

    /** Scheduled call: return keys with highest QPS in window */
    public List<HotKeyStat> topN(int n, long minQps) {
        return windows.entrySet().stream()
            .map(e -> new HotKeyStat(e.getKey(), e.getValue().qps()))
            .filter(s -> s.getQps() >= minQps)
            .sorted(Comparator.comparingLong(HotKeyStat::getQps).reversed())
            .limit(n)
            .collect(Collectors.toList());
    }

    /** Simple second-level sliding window: fixed-size ring buffer */
    static class SlidingWindow {
        private final AtomicLongArray buckets;
        private final int size;
        private volatile long lastSecond;
        private final AtomicLong windowSum = new AtomicLong();

        SlidingWindow(int size) {
            this.size = size;
            this.buckets = new AtomicLongArray(size);
            this.lastSecond = System.currentTimeMillis() / 1000;
        }

        void incr() {
            long sec = System.currentTimeMillis() / 1000;
            advance(sec);
            buckets.incrementAndGet((int) (sec % size));
            windowSum.incrementAndGet();
        }

        long qps() {
            advance(System.currentTimeMillis() / 1000);
            return windowSum.get() / size;
        }

        /** Clean expired buckets when second advances */
        private synchronized void advance(long now) {
            if (now <= lastSecond) return;
            long diff = Math.min(now - lastSecond, size);
            for (long i = 1; i <= diff; i++) {
                int idx = (int) ((lastSecond + i) % size);
                long old = buckets.getAndSet(idx, 0);
                windowSum.addAndGet(-old);
            }
            lastSecond = now;
        }
    }
}

Two details: the windows map only grows; the 100k cap is a safety net — add a scheduled cleanup for stale keys. Concurrency between advance and incrementAndGet is not atomic, causing minor statistical error, acceptable for hot-key detection.

Integration point warning : AOP on *Operations (e.g., ValueOperations, HashOperations) does not work because they are not Spring beans; the aspect silently fails. With Lettuce, register a CommandListener to capture command name and arguments:

client.addListener(new CommandListener() {
    @Override
    public void commandStarted(CommandStartedEvent event) {
        List<Object> args = event.getArgs();
        if (args.isEmpty()) return;
        Object first = args.get(0);
        if (first instanceof byte[]) {
            detector.record(new String((byte[]) first, StandardCharsets.UTF_8));
        }
    }
});

If you have a custom RedisClient wrapper, add counting there — don't try to proxy Spring's mappers.

Instrumentation overhead ~3-5% QPS drop, far lower than MONITOR. If still concerned, run full sampling only in staging/canary; production at 1% sampling.

Proxy Layer & MONITOR

If you run Codis, Twemproxy, or a custom proxy, count per-key at the proxy — zero intrusion, highest accuracy. Without a proxy, MONITOR is only for short diagnostics, e.g.,

timeout 10 redis-cli -h node monitor | grep -c "GET product:1001"

(stop after 10 seconds, never leave on production). OBJECT FREQ requires LFU policy and yields probabilistic decayed counts — only for rough screening.

Optimization: From Discovery to Elimination

Big Keys: Split, Compress, Async Delete

Structure splitting is most direct. A Hash with 1M fields: split into 100 keys via hash(userId) % 100, client routes by userId. Large List used as queue: split by time slice queue:20240101:00 or switch to Stream. Large String storing full JSON: split fields, high-frequency fields as separate Strings, full payload to DB. Large Set of tags: split by tag dimension into multiple Sets, combine with SINTER.

Hash sharding example:

public class BucketedHash {
    private static final int BUCKETS = 128;

    public void hset(String bizKey, String field, String value) {
        String bucket = bucketOf(bizKey, field);
        redis.opsForHash().put(bucket, field, value);
    }

    public Map<Object, Object> hgetAll(String bizKey) {
        // If full read is truly needed, use pipeline to fetch in parallel;
        // but most scenarios should fetch only required fields.
        List<Object> results = redis.executePipelined((RedisCallback<Object>) conn -> {
            for (int i = 0; i < BUCKETS; i++) {
                conn.hashCommands().hGetAll(
                    (bizKey + ":" + i).getBytes(StandardCharsets.UTF_8));
            }
            return null;
        });

        Map<Object, Object> all = new HashMap<>();
        for (Object r : results) {
            if (r instanceof Map) {
                all.putAll((Map<?, ?>) r);
            }
        }
        return all;
    }

    private String bucketOf(String bizKey, String field) {
        return bizKey + ":" + Math.floorMod(field.hashCode(), BUCKETS);
    }
}

After sharding, avoid hgetAll full pulls — 128 buckets concatenated still equals full data. Normal access should fetch by field; full reads only for offline/low-frequency via pipeline. Also field.hashCode() sharding only distributes key count; if a single field is hot, further split that field.

Compression : Application layer can use Snappy or LZ4 on values. Small objects: Hash uses less memory than String; Hash's ziplist encoding is very efficient when fields and values are small.

Deletion : Never use DEL on big keys. A 1M-element Hash DEL blocks main thread for seconds. Since Redis 4.0, use UNLINK — single command, background thread frees memory. Older versions: batch delete with HSCAN:

public void safeDelete(String key, int batch) {
    ScanOptions options = ScanOptions.scanOptions().count(batch).build();
    try (Cursor<Map.Entry<Object, Object>> cursor = redis.opsForHash().scan(key, options)) {
        List<Object> fields = new ArrayList<>(batch);
        while (cursor.hasNext()) {
            fields.add(cursor.next().getKey());
            if (fields.size() >= batch) {
                redis.opsForHash().delete(key, fields.toArray());
                fields.clear();
            }
        }
        if (!fields.isEmpty()) {
            redis.opsForHash().delete(key, fields.toArray());
        }
    }
    redis.delete(key);
}

Also enable lazyfree-lazy-expire, lazyfree-lazy-server-del, lazyfree-lazy-eviction so expiration and eviction run asynchronously. lazyfree-lazy-user-del is 6.0+ only; ignore if version lower.

Hot Keys: Local Cache, Replica Sharding, Read/Write Split

Ordered by cost:

Local cache (first choice). Caffeine absorbs 90%+ reads; 1-second TTL tolerable for most hot spots:

private final Cache<String, String> localCache = Caffeine.newBuilder()
    .maximumSize(10_000)
    .expireAfterWrite(1, TimeUnit.SECONDS) // tolerate 1s inconsistency
    .build();

public String get(String key) {
    return localCache.get(key, k -> redis.opsForValue().get(k));
}

Must pair with active invalidation: on business update, publish MQ broadcast to evict corresponding key across instances. 1-second window may still be too wide for inventory-like scenarios.

Replica sharding (second). Replicate one logical key into N copies key#0 to key#N-1, client reads randomly, writes fan out to all copies (sync or async via MQ):

private static final int COPY = 8;

public String getWithReplica(String key) {
    int idx = ThreadLocalRandom.current().nextInt(COPY);
    return redis.opsForValue().get(key + "#" + idx);
}

Traffic originally saturating one shard spreads across N slots. Typical for flash-sale inventory. Do not add {} hash tags to replica keys — that would force them into the same slot, defeating the purpose.

Read/write split (third). In a 3-primary/3-replica cluster, enable READONLY on client to read from replicas, offloading more pressure. Trade-off: replication lag; avoid for strong-consistency scenarios.

Memory Governance: Fragmentation, Eviction Policy, Watermarks

Metric Collection

First, gather data per node:

public class RedisMetricsCollector {
    public RedisMetrics collect(RedisConnection conn) {
        Properties info = conn.serverCommands().info();
        Properties mem = conn.serverCommands().info("memory");
        Properties stat = conn.serverCommands().info("stats");

        RedisMetrics m = new RedisMetrics();
        m.setUsedMemory(bytes(mem, "used_memory"));
        m.setUsedMemoryRss(bytes(mem, "used_memory_rss"));
        m.setMaxMemory(bytes(mem, "maxmemory"));
        m.setFragRatio(parseDouble(mem.getProperty("mem_fragmentation_ratio")));
        m.setEvictedKeys(parseLong(stat.getProperty("evicted_keys")));
        m.setExpiredKeys(parseLong(stat.getProperty("expired_keys")));
        m.setConnectedClients(parseInt(info.getProperty("connected_clients")));
        m.setBlockedClients(parseInt(info.getProperty("blocked_clients")));
        m.setOpsPerSec(parseLong(stat.getProperty("instantaneous_ops_per_sec")));
        return m;
    }
}

In cluster mode, call collectAll() iterating every node — don't sample just one.

Fragmentation Ratio

mem_fragmentation_ratio

< 1: RSS < logical memory → swap likely, critical, scale immediately.

1 – 1.5: normal.

> 1.5: significant fragmentation, consider action.

> 2: severe, OOM risk imminent.

Older Redis: enable activedefrag yes with active-defrag-threshold-lower 10 for online defrag. Newer versions: failover to replica, restart replica to reclaim memory, then fail back.

Choosing Eviction Policy

Ask: is this Redis pure cache or does it persist data?

Pure cache → allkeys-lru (simplest).

Clear hot/cold split, long tail → allkeys-lfu (resists scan pollution, needs 4.0+).

Mixed TTL / non-TTL keys → volatile-lru dangerous : keys without TTL never evicted; memory grows while evicted_keys stalls, eventually OOM. This