Redis‑On‑Rocks (ROR): Architecture, Cold‑Hot Data Swapping, and Performance Evaluation

Background

Redis stores data in memory, offering excellent performance but limited by memory capacity and cost. With modern NVMe SSDs providing microsecond‑level latency, using SSDs as an extension medium can lower cost while preserving low latency.

To address this, Ctrip developed Redis‑On‑Rocks (ROR), which augments the Redis kernel to support cold‑hot data swapping, expanding cache capacity onto disk and cutting overall cost by roughly two‑thirds.

ROR Overview

ROR builds on the Redis codebase and adds a cold‑hot multi‑level storage model. Hot data remains in Redis memory with native data structures; cold data is stored in RocksDB at sub‑key granularity.

The system handles two main operations:

Swap‑in: when a client accesses cold data, the required sub‑key is read from RocksDB and merged into Redis memory, after which command execution proceeds as in native Redis.

Swap‑out: when memory usage exceeds maxmemory , hot data is evicted to RocksDB using Redis’s LFU algorithm.

Because ROR inherits Redis’s data structures and command implementations, it is compatible with almost all Redis commands and can quickly adopt new Redis features.

Comparison with Redis‑on‑Flash (RoF)

Both ROR and RoF aim to reduce memory cost, but RoF stores keys in memory and values on SSD, leading to higher memory consumption for keys and less flexibility. ROR stores both keys and values in RocksDB, eliminating the need for in‑memory key metadata.

Key advantages of ROR over RoF:

Cost : ROR does not keep cold keys in memory, saving up to 72% of memory for values and an additional 29% for cold keys.

Generality : ROR leverages RocksDB’s table cache and OS page cache, narrowing latency gaps between hot and cold keys, making it suitable for workloads without clear hot‑cold patterns.

Performance : For hash‑type commands (e.g., HGET, HSET), ROR reads/writes only the needed sub‑keys, achieving 2‑3× higher QPS under heavy load and 300‑500 µs latency under normal load, far better than RoF’s 14‑120 ms.

Implementation Details

Cold‑Hot Swapping Process

The swap consists of five steps: syntax analysis, locking, submitting a swap task to an IO thread pool, performing RocksDB read/write, and merging the data back into Redis. After the swap, command execution continues exactly as in native Redis.

Concurrency Control

Since RocksDB operations are blocking, ROR uses a dedicated IO thread pool. To avoid data races, ROR implements a single‑thread re‑entrant lock per key, realized as a waiting queue. Clients requesting the same key are serialized, while different keys can proceed concurrently. Locks also exist at DB and server granularity for commands affecting the whole keyspace, and the lock is re‑entrant to support MULTI/EXEC transactions.

Cold Data Storage

Cold data is stored in RocksDB. Keys are kept in a meta column family (metaCF) containing type, expiration, version, and sub‑key count. Sub‑keys are stored as individual RocksDB key‑value pairs, enabling lazy deletion via compaction filters. Hash, set, and zset structures each have a meta entry and N sub‑key entries; lists are modeled as a combination of hot (memlist) and cold (rockslist) segments.

Cuckoo Filter for Miss Reduction

To avoid unnecessary RocksDB reads for non‑existent keys, ROR employs a cascading cuckoo filter (deletable, ~8 bits per key) which reduces miss‑related IO and improves QPS by about 20% in miss‑heavy scenarios.

Replication Compatibility

ROR’s replication protocol is fully compatible with native Redis. Full sync uses RDB format; incremental sync uses RESP. For cold data, ROR creates a RocksDB checkpoint and streams data into the RDB without loading all cold keys into memory, achieving ~315 MB/s (≈80% of pure Redis replication speed).

Production Experience

In Ctrip’s production environment, ROR instances host thousands of Redis instances, saving up to two‑thirds of memory cost and millions of RMB annually. Typical workloads see only a modest latency increase (≈20 µs) after migration, while QPS capacity remains well above the observed demand.

Open‑Source and Future Plans

ROR is open‑sourced under the BSD license (see GitHub ). Future work includes increasing single‑instance QPS, optimizing less‑used data structures (bitmap, stream), and reducing full‑sync overhead by allowing limited inconsistency for faster incremental sync.