How to Shrink 2 Billion DAU Stats from 8GB to 24MB with Redis Bitmap and Roaring

Step 1: The cost of using Set

Using a Redis Set to record 200 million user IDs appears simple but consumes far more memory than expected. The ideal calculation (8 bytes per ID plus ~16 bytes overhead) yields about 4.8 GB, while real‑world factors such as hash table load factor, jemalloc fragmentation, replication, and AOF/RDB persistence push memory usage to 6‑8 GB, which is unacceptable for a single DAU metric.

SADD daily_active:2023-10-01 userId

users = 200000000
userId = 8 bytes
Set overhead ≈ 16 bytes/element
Total ≈ 4.8GB (ideal) → real 6~8GB

Step 2: Using Bitmap for 200× compression

Redis Bitmap stores a bit per user, turning the 200 million entries into a 25 million‑byte (≈23.8 MB) array, a compression ratio close to 250× compared with Set.

SETBIT daily_active:2023-10-01 userId 1
BITCOUNT daily_active:2023-10-01

Memory comparison (text): Set 6~8 GB, Bitmap ≈24 MB – roughly 250× reduction.

Bitmap also supports bitwise operations for retention analysis.

# two‑day retention
BITOP AND retained daily_active:10-01 daily_active:10-02
BITCOUNT retained

Step 3: Bitmap is not always O(1)

Although BITCOUNT is often described as O(1), it actually scans the whole bitmap (size/8). For a 125 MB bitmap the scan touches ~15 MB of memory, blocking Redis’s single thread and affecting other commands, especially during RDB/AOF persistence.

Thread blockage

Impact on other requests

More severe during RDB/AOF

Therefore bitmaps must be sharded to keep each scan small.

Step 4: Sharded bitmap design

Shard the bitmap by user‑ID range, e.g., one shard per 1 million users. Example Java helper functions generate the shard key and bit position.

// each 1 M users one shard
public String shardKey(long userId, String date) {
    long shard = userId / 1_000_000;
    return "daily_active:" + date + ":shard_" + shard;
}
public long bitPosition(long userId) {
    return userId % 1_000_000;
}

Each shard bitmap is ~125 KB; with 200 shards the total memory is about 25 MB, and BITCOUNT only scans 125 KB, which is safe for the Redis thread.

Step 5: Continuous ID requirement

Bitmap works efficiently only when IDs are dense. Sparse identifiers such as UUIDs, Snowflake IDs, or random 64‑bit numbers would force Redis to allocate astronomic memory. SETBIT key 9223372036854775807 1 Solution: map real IDs to continuous numbers using an external store (MySQL, Redis Hash, or ClickHouse Dictionary).

Step 6: Roaring Bitmap for sparse data

When the active user set is a small fraction of the total (e.g., 1 million active out of 200 million), Roaring Bitmap reduces memory to 2‑3 MB versus 24 MB for a plain bitmap.

RoaringBitmap bitmap = new RoaringBitmap();
bitmap.add(userId);
bitmap.runOptimize();
RoaringBitmap retained = RoaringBitmap.and(today, yesterday);

Step 7: ClickHouse + Bitmap for analytical workloads

Redis handles low‑latency real‑time counting, while ClickHouse stores bitmap columns for large‑scale, complex analysis.

SELECT date,
       bitmapCardinality(bitmapBuild(groupArray(user_id))) AS dau
FROM user_activity
GROUP BY date;

SELECT bitmapAndCardinality(today_bitmap, yesterday_bitmap) AS retained
FROM ...;

Step 8: Real‑world three‑layer architecture

App → Kafka → Flink (deduplication & aggregation) → Redis Bitmap (7‑day real‑time DAU) → Roaring Bitmap persistence → ClickHouse Bitmap table → BI / retention / conversion analysis.

Step 9: Interview answer template

“I would never use a Set for 200 M DAU because it would explode Redis. I use a sharded Redis Bitmap, shrinking memory from 8 GB to ~24 MB; if IDs are sparse I upgrade to Roaring Bitmap; for multi‑dimensional analysis I delegate to ClickHouse’s bitmap support. This reflects a hot‑cold separation and compute‑storage decoupling architecture.”

Step 10: Design principles

Calculate memory consumption before writing code.

Redis should only handle real‑time data.

Large datasets must be sharded.

Sparse data must be compressed.

Analytical workloads belong in a columnar store.

Technology choices are always resource trade‑offs.