Apache Paimon 0.7.0: Enhanced Lookup Join, CDC Capabilities, and Spark/Hive Integration

Apache Paimon has released version 0.7.0, bringing a series of enhancements that strengthen its role in the data lake ecosystem and broaden its applicability in real‑time data development.

Look up Join

In streaming data pipelines, lookup join is often treated as a dimension‑table association. The new release adds several optimizations:

修复了lookup join 不能正确处理维表的 sequence field 问题。<br/>基于 Paimon 的 hash lookup join，添加了 primary key partial lookup 功能。<br/>通过并行读取文件和批加载的方式，加快了维表的初始化数据加载速度。<br/>

Although dimension‑table joins are frequently used in production, the author notes that Paimon/Hudi may not be the best choice for storing dimension data, recommending alternatives such as HBase or Redis, especially when dealing with large‑scale data and cache‑related challenges.

CDC Capability

The CDC (Change Data Capture) feature is split into two parts: first, ingesting CDC data into Paimon, which is becoming increasingly mature across lake‑table solutions; second, the native Paimon CDC capability, which promises future support for both batch and streaming reads and could reshape existing architectures.

Spark/Hive Integration

Support for Spark and Hive continues to improve, laying a foundation for broader adoption of lake‑table frameworks. Additional enhancements include the new level0FileCount metric for monitoring compaction progress and strengthened time‑travel capabilities.

Overall, the community encourages ongoing monitoring of Paimon’s development, as its features are expected to see wider and deeper use in production environments.