Apache Paimon Becomes a Top-Level Project: A Comprehensive Overview of Lakehouse Framework Capabilities and Future Trends

On April 16, 2024, the Apache Software Foundation announced that Apache Paimon has graduated to a Top-Level Project, marking a significant milestone for real‑time data lake and stream‑batch processing technologies.

The author presents a summary of the lakehouse landscape, emphasizing that frameworks such as Hudi, Paimon, Iceberg, and Delta aim to provide a unified "stream‑in‑store‑out" service. The essential capabilities identified are:

Streaming read/write with sub‑second latency and high‑throughput incremental consumption (targeting tens of millions of RPS, though current implementations fall short of Kafka‑level performance).

Batch read/write that matches Hive’s functionality while adding partitioned concurrent updates, primary‑key updates, and other advanced features.

Multi‑engine integration with Spark, Flink, Presto, etc., ensuring balanced support across processing engines.

Additional extensions such as changelog aggregation, external table mounting, and column‑level updates that surpass traditional data‑warehouse systems.

The article explains that lakehouse frameworks address specific shortcomings of traditional data warehouses, offering solutions for scenarios where existing pipelines are costly or insufficient, such as replacing parts of Kafka for analytical queries, decoupling storage and compute to reduce OLAP expenses, and enabling efficient primary‑key updates in batch workloads.

In conclusion, the lakehouse domain is rapidly maturing in major Chinese tech companies, and as framework capabilities grow, conventional data‑development models will increasingly be supplanted, heralding a new era of data engineering.