Paimon Project Overview: Recent Developments, Core Capabilities, and Future Roadmap

The presentation introduces Paimon, a real‑time lakehouse format that combines lake storage with LSM‑tree structures, enabling seamless integration with Flink and Spark for unified stream‑batch processing.

Core capabilities are organized into four parts: (1) an overview of Paimon’s architecture; (2) existing features including primary‑key tables (supporting upserts, CDC ingestion, bucket strategies, compression, merge engines, and changelog production) and log tables (offering queue‑like semantics, Z‑order indexing, and COW/MOR update mechanisms); (3) management utilities such as snapshot + tag versioning, system tables, procedures, and extensive metrics; and (4) a Q&A session addressing schema evolution, binary file handling, CDC integration, upgrade considerations, and performance tuning.

Performance enhancements focus on improving read speed for primary‑key tables by adjusting bucket sizes, employing deletion vectors to avoid costly merges, and optimizing Spark SQL via dynamic partition pruning, exchange reuse, adaptive scan concurrency, scalar sub‑query merging, and cost‑based optimization.

Future roadmap includes query acceleration for primary‑key tables, stream‑read changelog separation, log‑table query acceleration with bitmap/Bloom‑filter/inverted indexes, full CRUD support for log tables (COW and MOR), continued Spark integration, branch management (Git‑like branching, merging, and replacement), and broader catalog support (JDBC, REST).

The Q&A highlights practical guidance such as handling schema evolution in Flink SQL, binary file storage recommendations, upcoming Flink CDC 3.x sink, upgrade notes from 0.6 to 0.7, bucket configuration for log tables, and the distinction between lakehouse latency and traditional message queues.