RocksDB: An Introductory Look at a High‑Performance Key‑Value Store

In distributed systems and big‑data applications, an efficient and reliable storage engine is essential. RocksDB, created by Facebook in 2012 and later open‑sourced, is a persistent key‑value store designed for flash and high‑speed disks, targeting fast low‑latency access while maintaining high reliability and scalability.

Core Features

High performance : Optimized for fast storage devices using techniques such as multithreaded compaction writes, data compression and delayed deletion to achieve high read/write throughput.

Persistence : Guarantees data durability; data remains recoverable after crashes or restarts.

Tunability : Offers extensive configuration options that let developers tune performance for specific workloads.

Hardware support : Runs on SSD, HDD, NVMe and can exploit multi‑core processors for parallel data handling.

Compatibility : Supports multiple operating systems and programming languages, facilitating easy integration.

Internal Structure

MemTable : An in‑memory data structure that caches recent writes; when it reaches a size limit it is flushed to an on‑disk SSTable.

SSTable (Sorted String Table) : A persistent, key‑sorted file stored on disk; each SSTable contains a sequence of key‑value pairs ordered by key.

Write‑Ahead Log (WAL) : Records write operations before they reach the MemTable, ensuring durability and enabling recovery if a crash occurs before flushing.

Compaction : Merges multiple SSTable versions over time, discarding obsolete entries and reorganizing data to reduce space usage and improve read efficiency.

Bloom Filter : A probabilistic data structure that quickly checks whether a key may exist in an SSTable, avoiding unnecessary disk I/O.

Application Scenarios

Database systems : Serves as the underlying storage engine for relational or NoSQL databases.

Distributed systems : Provides fast local storage for services such as ZooKeeper‑backed coordination, ensuring consistency and availability.

Big‑data processing : Offers high throughput and low latency for frameworks like Hadoop and Spark.

Real‑time analytics and stream processing : Supplies rapid read/write capabilities required by low‑latency applications.

RocksDB in TiDB

TiKV, the storage layer of the distributed SQL database TiDB, uses RocksDB as its underlying engine. RocksDB’s key‑value interface and LSM‑tree architecture provide durable data persistence and efficient reads/writes, allowing TiKV to exploit multi‑CPU environments, fast SSD storage, and elastic scaling.

RocksDB in Apache Flink

Apache Flink adopts RocksDB as the default state backend. Its advantages include:

Local storage : State is kept on local disks, dramatically reducing access latency compared with networked file systems.

Efficient writes : Combines WAL and in‑memory MemTable for fast, durable writes.

Data compression : Supports algorithms such as Snappy and Zlib to save space and improve read speed.

Multi‑Version Concurrency Control (MVCC) : Allows concurrent readers and writers without interference, essential for Flink’s parallel processing.

Fault recovery : Persisted state enables recovery from node failures using backups from other nodes.

Scalability : Design permits easy expansion across multiple disks and nodes, maintaining high performance for large‑scale streams.