Row-Level Cross-Table MVCC for Lakehouses: Retina's 10ms Freshness at 5% Query Cost

Current lakehouse formats (Iceberg, Delta Lake, Hudi, Paimon, Lance) implement file-level catalog transactions rather than row-level MVCC. They only support serial writes — akin to git's serial commits — and ACID guarantees cover single tables at the file level, making cross-table transactions impossible. Two implementation approaches exist: Iceberg, Paimon, and Lance materialize each version as a catalog snapshot for direct reads, while Delta Lake and Hudi record write-behind logs and reconstruct snapshots at query time. The update path forces a trade-off: Copy-on-Write (CoW) favors reads but makes updates expensive; Merge-on-Read (MoR) suits CDC workloads but pushes cost to queries. Compensation mechanisms include runtime merging, delete files, delete vectors (DV), and compaction.

Benchmarking Five Formats

The team evaluated all five formats in a SIGMOD FORMATS'26 paper using HyBench SF1333 (1 TB, 116 billion rows) and CH-benCHmark WH10000 (778 GB, 5 billion rows) with a workload of ~70% random updates, 25% inserts, and 5% deletes. Results show Iceberg achieves good freshness but lower throughput than Paimon; Paimon delivers high throughput yet enabling delete vectors degrades write performance; Delta Lake builds delete vectors inefficiently; Lance and Hudi exhibit lower index efficiency than Paimon. Indexes across all formats face three common challenges: too large to fit in memory, random I/O unfriendliness, and the need to stay aligned with catalog and version metadata.

Fundamental Limits: Not True MVCC, Bound by RUM

Two core judgments emerge. First, existing lakehouses are not true MVCC — batch updates, serial writes, and single-table ACID cannot support row-level, cross-table, concurrent real-time replay. Second, they cannot escape the RUM (Read-Update-Memory) trade-off: query performance, update cost, and memory overhead pull against each other. Each format occupies a different corner of the RUM triangle; optimizing all three simultaneously requires structural change, not incremental patches.

Retina: A New Table Format Layer on Pixels

Retina, built atop the Pixels engine (PVLDB Vol. 19, 2026), introduces four innovations:

Cross-table mirror transactions: CDC messages arrive in source commit order. The sink groups them by transaction ID into mirror transactions, assigns monotonically increasing timestamps in arrival order, and hashes them to multiple replayers for parallel replay. A single source transaction remains a single transaction in the lakehouse.

Row-level visibility: Row groups are split into 4,096-row tiles. Each tile carries a visibility structure — versions before a base timestamp are recorded in a bitmap; deletions after that base are appended to a delete-timestamp array. As the low-water mark (LWM) advances, entries merge back into the bitmap for memory reclamation. Queries use SIMD-vectorized filtering with no concurrency conflicts against writes or garbage collection.

Row-level MVCC: All transactions before the high-water mark (HWM) have been successfully replayed; all queries before the LWM have completed. New queries snapshot at HWM-1, and every tuple version carries a create_ts and delete_ts.

No compaction: New versions enter a write buffer and are flushed in 4,096-row blocks to S3 as large files. Old files are partially rewritten only when the deleted-version ratio exceeds a threshold. This eliminates small-file problems and visibility-judgment overhead, making compaction unnecessary for scan-heavy queries. Indexes are implemented as plugins that share the same multi-version rules as data.

Results and Deployments

Retina achieves 10 ms data freshness versus 10 seconds to minutes for current lakehouses. High-throughput updates incur only ~5% query-performance degradation. Memory overhead stays below comparable lakehouse and HTAP systems, and throughput scales linearly to millions of rows per second. The visibility structure consumes 1.1–2 bits per tuple version — under 1 GB for a 1 TB dataset. The system is already deployed in China Mobile's Panwei OLAP, Tencent TDSQL, Huawei Munich Research Center, and Microsoft Bing's search-log data lake, with key optimizations incorporated into national standard GB/T 41818-2022.