Building a Real‑Time Lakehouse with StarRocks, Fluss, and Paimon: A Closed‑Loop ETL Architecture

Traditional Lambda architectures separate real‑time and batch pipelines, requiring multiple systems, duplicated storage, and complex consistency management, which hampers the growing demand for sub‑second data freshness and operational efficiency.

In the proposed lakehouse‑in‑one design, Flink CDC captures changes from business systems and writes them to Fluss; Fluss stores real‑time data and tiers older data to Paimon, which serves as the long‑term lakehouse storage. StarRocks unifies analysis and ETL by using Union Read to simultaneously query Fluss (real‑time) and Paimon (historical) data, and by executing lake‑side ETL whose results are written back to Paimon, creating a complete processing loop.

Union Read works by splitting a table into a historical segment (read from Paimon snapshots) and a real‑time segment (read from Fluss offsets). A single SQL query therefore returns both the latest incremental rows and the full historical view, eliminating the batch‑vs‑stream split.

StarRocks offers three read modes for the same logical table: default (Union Read) for full freshness, table$lake to read only Paimon (lower cost, stable snapshots), and table$rt to read only Fluss (real‑time only). This flexibility lets users balance completeness, latency, and cost per query.

Reading performance benefits from native C++ access: Paimon Native reads avoid JVM overhead, while Fluss initially used JNI for compatibility. Over time, Fluss reading also migrated to native, reducing cross‑language calls and memory copying.

Benchmark results on a 4‑node BE cluster (8 CPU/32 GB each) show that Native writes to Paimon are 1.7×‑5.1× faster than JNI, with the 100 GB workload completing only with Native (JNI OOM). Native reads outperform JNI by 1.4×‑3.4× across point‑lookup, filter, and wide‑table aggregation queries. Similar gains appear for Fluss reads, where Native reduces Scan time by 2.0×‑2.7× and overall query latency by 1.4×‑1.8×.

A concrete use case is real‑time risk control: a suspicious transaction can be re‑validated in a single SQL that pulls the latest logs from Fluss and the historical profile from Paimon, delivering sub‑second decision latency.

Future work focuses on “read faster, write more fully”: extending Union Read to support Deletion Vectors for MOR tables, enriching the optimizer with row‑count and NDV statistics, expanding native support to PK tables, completing Paimon schema evolution, and integrating lake‑wide materialized view rewrites.

Overall, the combination of StarRocks, Fluss, and Paimon provides a unified, high‑performance lakehouse that supports real‑time analytics, scalable ETL, and seamless result feedback, while reducing operational complexity and data duplication.