How to Efficiently Count 100 Million Redis Keys Without Crashing Your Cluster

Introduction

Many developers encounter a scenario where a boss asks to count all keys in Redis, and using the KEYS * command can freeze the entire cluster.

This article explores efficient ways to count up to 100 million Redis keys.

1 Why Not Use KEYS?

Redis uses a single‑threaded model, so KEYS * must traverse the entire keyspace (O(N)) and blocks all other commands. For 100 M keys, even a 0.1 µs lookup per key takes over 10 seconds, leading to CPU, memory, and cluster failures.

Time complexity : 10 seconds+ for 100 M keys.

Memory storm : Result set may exhaust client memory.

Cluster failure : In Cluster mode, only the current node’s data is scanned. 127.0.0.1:6379> KEYS * Result: OOM error when memory limit is exceeded.

2 SCAN Command

The SCAN command iterates with a cursor, returning a small batch of keys each time, avoiding blocking.

public long safeCount(Jedis jedis) {
    long total = 0;
    String cursor = "0";
    ScanParams params = new ScanParams().count(500); // 500 per batch
    do {
        ScanResult<String> rs = jedis.scan(cursor, params);
        cursor = rs.getCursor();
        total += rs.getResult().size();
    } while (!"0".equals(cursor));
    return total;
}

Scanning 100 M keys at 3 ms per batch of 500 requires about 20 000 batches, roughly 10 minutes.

3 Multithreaded SCAN

Modern servers have multiple CPU cores, so a single‑threaded scan wastes resources. Below is a parallel SCAN implementation using a thread pool.

public long parallelCount(JedisPool pool, int threads) throws Exception {
    ExecutorService executor = Executors.newFixedThreadPool(threads);
    AtomicLong total = new AtomicLong(0);
    List<String> cursors = new ArrayList<>();
    for (int i = 0; i < threads; i++) {
        cursors.add(String.valueOf(i));
    }
    CountDownLatch latch = new CountDownLatch(threads);
    for (String cursor : cursors) {
        executor.execute(() -> {
            try (Jedis jedis = pool.getResource()) {
                String cur = cursor;
                do {
                    ScanResult<String> rs = jedis.scan(cur, new ScanParams().count(500));
                    cur = rs.getCursor();
                    total.addAndGet(rs.getResult().size());
                } while (!"0".equals(cur));
                latch.countDown();
            }
        });
    }
    latch.await();
    executor.shutdown();
    return total.get();
}

Performance comparison (32‑core CPU, 100 M keys): single‑threaded SCAN takes ~580 s (CPU 5%), multithreaded SCAN with 32 threads takes ~18 s (CPU 800%).

4 Distributed Counting in Cluster Mode

In Redis Cluster, data is sharded across 16 384 slots. Each master node can count its assigned slots, and results are aggregated.

public long clusterCount(JedisCluster cluster) {
    Map<String, JedisPool> nodes = cluster.getClusterNodes();
    AtomicLong total = new AtomicLong(0);
    nodes.values().parallelStream().forEach(pool -> {
        try (Jedis jedis = pool.getResource()) {
            if (jedis.info("replication").contains("role:slave")) return;
            String cursor = "0";
            do {
                ScanResult<String> rs = jedis.scan(cursor, new ScanParams().count(500));
                total.addAndGet(rs.getResult().size());
                cursor = rs.getCursor();
            } while (!"0".equals(cursor));
        }
    });
    return total.get();
}

5 Millisecond Counting Options

Option 1: Built‑in Counter

Use INFO keyspace to get an approximate count instantly (O(1)), but it includes expired keys.

127.0.0.1:6379> info keyspace
# Keyspace
db0:keys=100000000,expires=20000,avg_ttl=3600

Option 2: Real‑time Incremental Counting

Subscribe to keyspace notifications and maintain an exact counter.

@Configuration
public class KeyCounterConfig {
    @Bean
    public RedisMessageListenerContainer container(RedisConnectionFactory factory) {
        RedisMessageListenerContainer container = new RedisMessageListenerContainer();
        container.setConnectionFactory(factory);
        container.addMessageListener((message, pattern) -> {
            String event = new String(message.getBody());
            if (event.startsWith("__keyevent@0__:set")) {
                redisTemplate.opsForValue().increment("total_keys", 1);
            } else if (event.startsWith("__keyevent@0__:del")) {
                redisTemplate.opsForValue().decrement("total_keys", 1);
            }
        }, new PatternTopic("__keyevent@*"));
        return container;
    }
}

This method provides precise counts with modest memory and CPU overhead.

6 How to Choose a Solution?

Consider time, space, and accuracy requirements:

Solution

Time Complexity

Space Complexity

Accuracy

KEYS

O(N)

O(N)

Exact

SCAN

O(N)

O(1)

Exact

Built‑in Counter

O(1)

O(1)

Inexact

Incremental Counter

O(1)

O(1)

Exact

Guidelines:

CPU‑bound: threads = CPU cores × 1.5

IO‑bound: threads = CPU cores × 3

Control batch size (COUNT) to stay within memory limits.

Typical scenarios:

E‑commerce dashboards – incremental counter + RedisTimeSeries.

Offline analytics – SCAN export to Spark.

Security audits – parallel SCAN across nodes.

Final advice: Use divide‑and‑conquer for precise counts, incremental counters for real‑time queries, sampling for trend analysis, and avoid full KEYS scans.