How to Scale Redis for Billions of Keys: Memory‑Efficient Strategies

When designing a real‑time data warehouse for a DMP that must serve billions of third‑party IDs (cookies, IMEI, IDFA) and their demographic tags, traditional HDFS storage is insufficient for the required millisecond‑level query latency.

Requirement Background

The DMP needs to manage massive supperid mappings, demographic tags, mobile IDs, blacklists, and IP data. Offline storage of trillions of records is easy with HDFS, but real‑time lookup demands in‑memory solutions.

Storing over 5 billion key‑value pairs consumes more than 1 TB of RAM, and high availability with replicas further inflates memory usage. Variable key lengths cause fragmentation, making a large‑scale storage design essential.

Data Stored

Keys include cookie, imei, idfa and values contain demographic fields such as gender, age, geo. Mapping relationships link media cookies to supperid.

Example storage formats:

PC ID:

媒体编号-媒体cookie=>supperid
supperid=>{age=>年龄段编码, gender=>性别编码, geo=>地理位置编码}

Device ID:

imei or idfa=>{age=>年龄段编码, gender=>性别编码, geo=>地理位置编码}

Data Characteristics

Short keys and values (e.g., 21‑digit supperid, MD5‑hashed IMEI, hyphenated MD5 IDFA).

Highly variable cookie lengths.

Scale: hundred‑billion supperid, trillion‑level media mappings, tens of billions of mobile IDs.

Billions of new mappings generated daily.

Only a fraction of data can be predicted as hot; many new IDs are completely cold.

Technical Challenges

1) Variable key lengths cause memory fragmentation. 2) Pointer‑heavy structures inflate memory up to 7×. 3) Massive daily influx of new IDs prevents effective hot‑data prediction. 4) Service latency must stay under 100 ms on public networks, requiring all new mappings to reside in memory. 5) Data retention of at least 35 days. 6) Memory cost for hundred‑billion‑key storage is prohibitive.

Solutions

Eviction Strategy

Implement TTL‑based cleanup: aggregate logs in HBase, set a 35‑day TTL, and configure Redis keys with a 35‑day expiration that is refreshed on access, effectively removing stale IDs.

Reducing Memory Bloat

Store key1=>value1 using a fixed‑length bucket identifier derived from an MD5 hash. The bucket ID becomes the Redis key, and the original key/value pair is kept inside a hash map.

Lookup flow: get(key1) -> hget(md5(key1)).

By allowing multiple keys to collide within a bucket (e.g., ~10 keys per bucket), the total number of Redis keys can be reduced by over 90%.

Implementation example (Java):

public static byte[] getBucketId(byte[] key, Integer bit) {
    MessageDigest mdInst = MessageDigest.getInstance("MD5");
    mdInst.update(key);
    byte[] md = mdInst.digest();
    byte[] r = new byte[(bit-1)/7 + 1]; // each byte holds 7 usable bits (ASCII 0‑127)
    int a = (int) Math.pow(2, bit%7) - 2;
    md[r.length-1] = (byte)(md[r.length-1] & a);
    System.arraycopy(md, 0, r, 0, r.length);
    for(int i=0;i<r.length;i++) {
        if(r[i] < 0) r[i] &= 127;
    }
    return r;
}

The bit parameter determines bucket space size; for 2³⁰ buckets (≈1 billion) each bucket can hold ~10 keys, supporting hundred‑billion‑scale storage.

Reducing Fragmentation

Make Redis keys fixed‑length by using the bucket ID and truncating device IDs to their last six characters for the hash‑map key. Values store only three‑byte codes for age, gender, and geo, ensuring memory alignment.

Additional tricks: periodically restart Redis slaves and trigger failover to let the master compact memory.

Recommended memory allocators: Google tcmalloc or Facebook jemalloc, which reduce fragmentation for small values compared to the default libc allocator.

By applying these strategies, the DMP can achieve low‑latency, real‑time queries while keeping memory consumption within feasible limits.