How StarRocks’ Real‑Time Storage Engine Evolves to Meet Modern Analytics Demands

1. Real‑time Update Storage Engine

About a year ago StarRocks released version 1.9 with a new storage engine that supports real‑time updates using a Delete + Insert (merge‑on‑read) approach. Compared with the previous merge‑on‑read uniq model, query performance improves 3‑10× while import performance remains almost unchanged. The engine is well‑suited for TP‑to‑AP real‑time synchronization, typically via Apache Flink CDC, and has been adopted by many production users.

2. Limitations of the Primary‑Key Model

The original design used an in‑memory primary‑key index for simplicity and high performance, but it consumes a large amount of memory, limiting its use to scenarios with controllable row counts or hot‑data partitions. High‑frequency imports cause “too many versions” and “too many pending versions” errors because the system was built for batch, large‑transaction workloads rather than many small, concurrent transactions generated by Flink CDC. Functional gaps also exist: lack of partial‑column updates, conditional updates, and comprehensive DML (update/delete/merge) support, as well as limited materialized‑view capabilities.

3. Recent Work

Persistent Primary‑Key Index : Implements a two‑level LSM‑like design—an in‑memory mutable hash table and an immutable disk‑based hash table. Memory usage drops to about one‑tenth of the original (BE process memory ~70‑80 GB, index memory ~3‑4 GB). SSD/NVMe is recommended for optimal performance.

Partial Column Update : Allows updating only specified columns, reducing data transfer. The current implementation requires all rows in a transaction to update the same set of columns. The workflow involves reading the existing row, merging the new column values, and performing a full‑row upsert with optimistic conflict handling.

Conditional Update : Supports simple conditions (e.g., src.ts > dest.ts) to prevent out‑of‑order overwrites. Only this expression is available now, with plans to add more complex conditions.

High‑Frequency Import Optimization : Parallelizes publish tasks and RocksDB metadata commits, introduces vertical‑compaction, and adds a streamload‑based transaction interface that merges multiple Flink sink tasks into a single transaction, dramatically reducing transaction count.

DML Support : Adds update and delete statements with read‑write transaction semantics (currently read‑committed isolation). Future work includes supporting more complex DML and eventually MERGE INTO syntax.

4. Future Plans

Separate Primary‑Key and Sort‑Key : Decouple the primary key (uniqueness and update target) from the sort key (storage order) to enable better query acceleration for non‑primary‑key filters.

Row‑Store Introduction : Explore adding a row‑store (or hybrid row‑column) layer to mitigate read‑write amplification in ultra‑wide tables, especially for partial‑column updates.

Materialized View Enhancements : Build a next‑generation materialized‑view architecture that supports transparent query acceleration, offline full builds, real‑time incremental builds, and requires binlog‑like change logs and persistent state storage.