Real-time Big Data Analytics with Apache Paimon and the Streaming Lakehouse Architecture

The article is compiled from Wang Feng's talk at the Streaming Lakehouse Meetup (May 16), covering five key topics: the concept of Data Lake + Data Warehouse = Data Lakehouse, Apache Paimon as a unified lake format, the past, present and future of Apache Paimon, the emergence of Streaming Lakehouse, and Paimon's designation as Alibaba's unified data lake format.

Lakehouse combines the strengths of data lakes and data warehouses, allowing unified storage of structured, semi‑structured, and unstructured data, and seamless integration with batch processing, streaming, OLAP, and machine‑learning/AI workloads.

To achieve real‑time analytics on a Lakehouse, two conditions are required: a real‑time compute engine and a data format that supports continuous updates. Existing lake formats such as Iceberg, Hudi, and Delta Lake are batch‑oriented and lack native real‑time update capabilities.

Apache Paimon addresses this gap by offering a streaming‑first storage format that supports low‑latency updates, CDC semantics, and batch operations, and integrates smoothly with engines like Apache Flink, Spark, Trino, Presto, and StarRocks.

Paimon originated from the Flink Table Store project, was incubated in the Apache Flink community, and later graduated to a top‑level Apache project in 2023, achieving integration with major analytics engines and planning a 1.0 release that unifies streaming, batch, and OLAP analysis while remaining compatible with formats like Iceberg.

The Streaming Lakehouse concept leverages Flink + Paimon to create an end‑to‑end real‑time data pipeline, reducing latency from hour‑level to seconds and enabling a true streaming Lakehouse architecture.

Within Alibaba, Paimon has been adopted as the unified data lake format across products such as Flink, Spark, StarRocks, MaxCompute, and Hologres, and will be offered as a cloud solution to help enterprises perform real‑time analytics at scale.

Finally, the presenter thanks the audience and encourages continued engagement with the Apache Paimon community.