Apache Paimon: Real‑Time Lakehouse Architecture, Core Technologies, Application Scenarios, and Frontier Features

The presentation introduces the emerging concept of a real‑time lakehouse, explaining why traditional batch‑oriented data warehouses suffer from latency and how integrating streaming and batch processing can achieve minute‑level data freshness.

It then reviews related technologies, starting with Apache Iceberg as a shared database storage format and highlighting its capabilities (object‑store friendliness, ACID transactions, DML support, time travel, schema evolution, tags, and branches). Apache Paimon builds on Iceberg’s ideas, adding native support for primary‑key upserts and stream‑style updates.

Paimon’s core innovation is the adoption of a Log‑Structured Merge‑Tree (LSM) storage layout within a lake format, enabling efficient incremental writes, low write amplification, and fast merge‑on‑read queries. The article also describes the “merge‑on‑write” approach that uses deletion vectors to achieve fast OLAP queries while maintaining updatable tables.

Several real‑world application scenarios are showcased: (1) CDC pipelines that write MySQL changes directly into Paimon for simplified, automated ingestion; (2) Alibaba’s intelligent engine that uses Paimon as a unified mirror of business databases, providing minute‑level streaming and batch access with reduced load on source systems; (3) Ant Group’s UV/ PV computation that replaces heavyweight Flink state with Paimon upserts, cutting CPU usage by 60% and improving checkpoint stability; (4) OLAP workloads where data is written to Paimon, optionally sorted or clustered, and then queried via Doris or StarRocks, achieving near‑OLAP latency at a fraction of the storage cost.

The frontier section outlines recent Paimon enhancements: native tag and branch support with automatic TTL, enabling Git‑like workflows for testing and validation; branch‑based isolation of streaming and batch workloads; and ongoing development of generic indexes (bitmap, inverted, Bloom filter) to boost data‑skipping and query performance in object‑store environments.

Finally, the article mentions the OpenLake initiative on Alibaba Cloud, which aims to provide a serverless, low‑cost, low‑latency real‑time‑batch unified solution built on Paimon, and invites readers to try Flink + Paimon or join the Apache Paimon community.