How Real‑Time Lakehouse and Apache Paimon Transform Modern Data Architecture

Real‑Time Lakehouse

Traditional batch warehouses (Hive + Spark/Hadoop) suffer from poor timeliness, both in ETL (hours‑to‑days) and query latency (minutes). Real‑time lakehouses aim to upgrade this model by unifying batch and stream processing on a single storage layer, achieving minute‑level or even second‑level data freshness.

Typical enterprise architectures separate batch (Hive, Spark) and real‑time (Flink, Kafka) pipelines, leading to high complexity and cost. Attempts such as the Kappa architecture or stream‑batch hybrid often encounter issues like high development difficulty, expensive resources, and incompatible storage formats.

Lake formats provide ACID guarantees, file‑level updates, and can support both batch and streaming reads, dramatically improving timeliness while keeping storage costs low.

Apache Paimon

Built on the foundations of Apache Iceberg, Paimon adds native support for primary‑key upserts, enabling true streaming updates without the delete‑then‑insert pattern. It stores data using a Log‑Structured Merge‑Tree (LSM), which reduces write amplification and allows efficient compaction and merge‑on‑read queries.

Paimon also introduces advanced features such as tags, automatic TTL for tags, branches (allowing separate stream and batch branches on the same table), and merge‑on‑write semantics that generate deletion vectors during writes for fast OLAP queries.

Future work includes bitmap and inverted indexes to further accelerate data‑skipping and query performance on object storage.

Application Scenarios

CDC ingestion: Flink writes CDC data into Paimon, Spark queries it, and compaction is handled automatically.

Unified business‑database mirror: Paimon serves as a low‑cost, minute‑level refreshed mirror of MySQL tables, reducing load on the source database and enabling both streaming and batch analytics.

Ant UV/ PV calculation: Using Paimon’s upsert and changelog, duplicate removal and real‑time metrics are achieved with 60% lower CPU usage and faster checkpoint recovery.

OLAP integration: Data written to Paimon can be sorted, clustered, and indexed, then queried efficiently by Doris or StarRocks, offering near‑OLAP performance at a fraction of the cost.

Frontier Technologies

Paimon now supports tag and branch management similar to Git, enabling isolated testing and versioned data pipelines. Merge‑on‑write combines the benefits of low‑latency reads with acceptable write performance, while ongoing work on bitmap and inverted indexes promises further query acceleration.