How ROR (Redis‑On‑Rocks) Cuts Cache Costs by 2/3 with Hot‑Cold Data Swapping

Background

Redis stores all data in memory, which gives low latency (≈100‑200 µs) but makes the cache expensive because memory is costly and limited. Modern NVMe SSDs provide random‑read latency of only a few tens of microseconds, allowing a disk tier to be used without a large latency penalty.

ROR Overview (Redis‑On‑Rocks)

ROR extends the Redis codebase with a multi‑level hot‑cold storage model:

Hot tier: Native Redis data structures reside in RAM.

Cold tier: RocksDB stores data on SSD at subkey granularity.

When a client accesses a cold key, ROR swaps the required subkeys from RocksDB into Redis memory; when memory pressure exceeds maxmemory, hot keys are evicted to RocksDB using Redis’s LFU algorithm. This reduces overall cache cost by roughly two‑thirds while keeping latency within acceptable bounds.

Comparison with Redis‑on‑Flash (RoF)

Both ROR and RoF add disk‑backed storage to Redis, but their designs differ:

Key placement: RoF keeps keys in memory and values on disk, leading to higher memory consumption. ROR stores both keys and values in RocksDB, eliminating the memory overhead of cold keys.

Cache utilization: ROR leverages RocksDB’s table cache and the OS page cache, while RoF disables these caches, causing larger latency gaps between hot and cold keys.

Performance: In high‑load benchmarks, ROR’s HGET/HSET QPS is 2‑3× that of RoF, and latency under normal load is 300‑500 µs versus RoF’s 14‑120 ms.

Implementation Details

1. Hot‑Cold Swap Process

Syntax analysis: Parse the incoming command to determine which keys and subkeys are required (e.g., MGET k1 k2 touches k1 and k2; HMGET h1 f1 f2 touches h1.f1 and h1.f2).

Lock acquisition: Obtain a custom single‑thread re‑entrant lock for each involved key to serialize I/O for that key.

Swap task submission: Dispatch a swap job to an I/O thread pool.

RocksDB I/O: Perform the necessary read or write against RocksDB.

Merge: Integrate the fetched subkeys back into Redis’s in‑memory data structures.

After the swap, the command proceeds exactly as in vanilla Redis.

2. Concurrency Control

RocksDB I/O runs on separate threads, so ROR introduces a single‑thread re‑entrant lock queue to guarantee that only one client manipulates a given key at a time. Locks exist at three granularities (key, DB, server) and are re‑entrant to support MULTI/EXEC transactions, preventing data corruption while preserving Redis’s single‑thread command ordering.

3. Cold Data Storage

Cold data is stored in RocksDB with three design points:

Keys are kept in a dedicated meta column family (type, expire, version, subkey count).

Each subkey maps one‑to‑one to a RocksDB key‑value entry, enabling subkey‑level swap‑in.

Deletion of aggregate types (hash, set, zset, list) is lazy: only the meta entry is removed; the actual subkeys are reclaimed later by RocksDB compaction using a compaction filter.

Encoding formats:

Hash/Set: One meta KV in metaCF plus one KV per subkey in defaultCF. The subkey KV stores the version to support lazy deletion.

Zset: Same as hash/set, with an additional scoreCF to store scores.

List: Represented as a sequence of segments; each segment is either a memlist (hot) or rockslist (cold). Segment metadata is stored in listObjectMeta.segments, and each element of a rockslist is a RocksDB KV.

4. Cuckoo Filter for I/O Reduction

Because cold keys are not kept in memory, every miss requires a RocksDB lookup to confirm non‑existence. ROR uses a cascading cuckoo filter (supports deletions) that consumes ~8 bits per key. This filter eliminates unnecessary reads for non‑existent keys, improving QPS by ~20 % in miss‑heavy workloads.

5. Replication Compatibility

ROR fully supports Redis replication protocols:

Full sync: A child process streams an RDB file. Hot data is serialized as usual; cold data is generated by taking a RocksDB checkpoint and scanning it, converting entries to RDB format on the fly. This stream‑based approach avoids loading cold keys into memory and achieves ~315 MB/s (≈80 % of native Redis).

Incremental sync: Commands received from the master are distributed to multiple worker threads for RocksDB I/O, but execution is deferred until all preceding commands have completed, preserving order and consistency.

Production Experience

In large‑scale deployments (e.g., Ctrip), memory usage typically reaches 50 % while CPU stays around 15 %. Adding ROR allows three Redis instances to share one ROR instance, cutting hardware cost by about two‑thirds. Thousands of ROR instances have been deployed, saving millions of dollars annually.

Latency impact is minimal: a migration with 80 % cold data increased the 95th‑percentile latency from ~200 µs to ~220 µs.

Open‑Source Release & Future Plans

ROR is released under the BSD license. Planned improvements include:

Increasing single‑instance QPS by reducing swap overhead and optimizing pipelining.

Adding support for less‑used data structures (bitmap, stream).

Providing optional eventual‑consistency modes to reduce full‑sync traffic for overseas deployments.