Paimon: Unifying Storage for Real-Time Data Warehouses

01 Paimon Architecture: Unifying Tables, Streams, and History

Apache Paimon is not a compute engine nor a traditional database, but a table storage format for data lakes. It decouples compute from storage: engines like Flink, Spark, Trino, StarRocks, and Doris read and write Paimon tables via Catalog and Connector, while data files reside on HDFS, S3, OSS, or other object stores. This allows a single copy of data to serve continuous writes, incremental consumption, offline backfill, and interactive queries without replication.

Table Organization

A Paimon table is organized through layered metadata:

Snapshot : Records table state at a point in time, linking current schema and Manifest List; enables time travel, rollback, and consistent reads.

Manifest List & Manifest : Track added/removed data, changelog, and index files per snapshot, with partition statistics for query planning and data skipping.

Partition & Bucket : Partition prunes by business dimension; Bucket is the fundamental unit for primary-key tables and LSM organization.

Data File & Changelog File : Data files hold current table data; changelog files provide incremental changes for downstream consumers.

Writes use two-phase commit to atomically produce new snapshots. Readers see only fully committed state. Concurrent writes across partitions can proceed in parallel; writes to the same partition receive snapshot isolation per official docs. Real-time warehouses should still define clear primary writers and partition boundaries.

Primary-Key Tables and LSM Updates

Traditional lake tables excel at appends but struggle with high-frequency upserts. Paimon introduces an LSM tree per bucket: new data lands in Level 0 files, then compacts incrementally. This avoids random object-store modifications. Paimon 2.0 offers three read-write trade-offs: MOR (Merge On Read): Fastest writes; reads merge multiple sorted files. Suits write-heavy, read-light workloads. COW (Copy On Write): Full merge on write; fastest reads but high write amplification. Suits read-heavy, write-light workloads. MOW (Merge On Write): Writes generate Deletion Vectors to mark stale rows; reads filter them out. Recommended as the general-purpose primary-key mode in current docs.

Thus Paimon makes write throughput, query performance, and data visibility configurable storage strategies.

Three Table Types for Three Data Shapes

Append Table : No primary key, append-only. Fits logs, detail facts, batch ETL, large-scale OLAP. Supports snapshots, time travel, statistics, file indexes, and incremental clustering.

Primary Key Table : Supports upsert, delete, and changelog. Fits database CDC, deduplication, wide tables, real-time aggregation.

Multimodal Table (Paimon 2.0): Extends Append Table with BLOB, vector, full-text, and global indexes for images, audio/video, documents, and AI data.

For real-time warehouses, Append Table handles immutable facts; Primary Key Table handles evolving business entities.

02 Core Capabilities and Application Scenarios

Scenario 1: Database CDC into Lake ODS

MySQL/PostgreSQL binlog produces continuous inserts, updates, deletes. Paimon Primary Key Table stores latest entity state and emits incremental changes downstream. Compared to "Kafka for changes + Hive for results", a single table supports both batch queries of current state and continuous reads from any snapshot, simplifying recovery, replay, and offline validation.

Scenario 2: Multi-Stream Real-Time Wide Tables

Orders, payments, logistics, user profiles arrive from different streams at different times. Paimon's partial-update Merge Engine lets each stream update only its owned fields; Sequence Group handles per-stream out-of-order events. This avoids waiting for all fields or keeping massive Flink join state on compute nodes.

Scenario 3: Storage-Side Real-Time Aggregation

For metrics like PV, revenue, counters, the aggregation Merge Engine performs sum, max, last_non_null_value per primary key during compaction. Streaming jobs only write increments; Paimon maintains results. It doesn't replace complex windowing but fits commutative, mergeable metrics, shifting recovery pressure from Flink state to durable table storage.

Scenario 4: Deduplication and First-Row Retention

Default deduplicate keeps latest value by key; first-row keeps first occurrence for log deduplication or first-touch attribution. Embedding semantics in the table's Merge Engine avoids duplicate logic in every downstream job.

Scenario 5: Unified Streaming-Batch Warehouse Layers

A single Paimon table supports three read modes: batch read of a historical snapshot, continuous streaming from latest position, incremental read from a specified snapshot. This enables collaboration:

Flink handles CDC, real-time cleansing, continuous derivation.

Spark handles historical backfill, recomputation, large-scale batch.

Trino, StarRocks, Doris handle interactive queries and serving.

Snapshots, Tags, and rollback manage release, audit, and recovery.

Scenario 6: Balancing Low Cost and Queryability

Data files default to Parquet, inheriting object-store cost efficiency and columnar scan benefits. Manifest statistics enable file pruning; Append Tables add Bloom Filter, Bitmap, Range Bitmap, Z-Order, Hilbert, or sort-based incremental clustering for layout optimization. Paimon doesn't turn object storage into a millisecond database but narrows the gap between continuous updates and queryability on low-cost lake storage.

Changelog: Underestimated and Misused

The changelog-producer option controls what downstream sees: none (default): Emits cross-snapshot merged changes only; lowest cost but downstream may need to retain old values. input: Preserves upstream full changelog; ideal for database CDC or when Flink already produces complete retract streams. lookup: During compaction, queries old values to generate full changelog; suits upstream with only new values but downstream needing Update Before. full-compaction: Generates changelog from full merge diff; accepts higher latency but incurs higher resource cost.

Full changelog is not free—it adds files, compaction, and storage overhead. Design should confirm downstream truly needs old values before enabling heavy modes.

03 Reference Architecture Highlighting Paimon Strengths

Treating Paimon as merely an upgraded Hive table wastes its primary-key updates and incremental consumption. A tailored real-time warehouse architecture:

ODS: Replayable Business Facts

Business databases write via Flink CDC into Paimon Primary Key Tables. With full CDC upstream, use changelog-producer = input; general primary-key tables should evaluate Deletion Vector mode for balanced write/query performance. Append-only events (Kafka, logs, tracking) go to Append Tables. ODS focuses on preserving complete facts, schema evolution, and replayable history.

DWD: Sinking Cross-Stream State into Tables

Flink reads ODS incremental snapshots or changelogs for cleansing, dimension enrichment, and wide-table building. For multi-source async arrival, use partial-update to merge fields; for simple latest-value semantics, use default deduplicate. The key shift: wide-table long-term state moves from Flink State to a multi-engine shareable, replayable, governable Paimon table.

DWS: Push-Down Aggregations to Storage

Simple mergeable metrics use aggregation tables; complex windows, session, CEP remain in Flink. DWS outputs feed both ADS and batch reuse.

ADS and Serving: Paimon Need Not Handle All Queries

Low-frequency reports and offline analysis run via Spark/Trino directly on Paimon; high-concurrency sub-second BI or online APIs use StarRocks/Doris reading Paimon or build serving-layer materialized views. Good architecture uses Paimon as a unified, reliable fact base, reducing duplicate storage and reconciliation.

Primary Key Table Example

Common ODS primary-key table configuration (note input only when upstream provides full CDC):

CREATE TABLE ods_order (
  order_id BIGINT,
  user_id BIGINT,
  order_status STRING,
  amount DECIMAL(18, 2),
  update_time TIMESTAMP(3),
  PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
  'bucket' = '-1',
  'deletion-vectors.enabled' = 'true',
  'changelog-producer' = 'input'
);

Four Pre-Launch Questions

Is this table append-only facts or continuously updated by primary key?

Does downstream need only latest values or full Update Before/After?

Priority: write throughput, query performance, or lower data visibility latency?

Who owns bucket sizing, compaction, snapshot retention, and small-file management?

Without answers, switching lake formats only defers complexity.

04 Quick Comparison with Other Storage Systems

Paimon, Iceberg, Hudi, Delta Lake are lake table formats with different design centers; Kafka and Fluss are streaming stores, not directly comparable on the same axis.

Paimon : Standout capability — Flink-native streaming writes, primary-key LSM, incremental reads, multiple Merge Engines. Best-fit scenarios — CDC into lake, real-time warehouse layering, wide tables, streaming aggregation. Boundary — Data visibility depends on checkpoint, commit, compaction; not a sub-millisecond online DB.

Iceberg : Standout capability — Neutral open table format, reliable snapshots, hidden partitioning & partition evolution, multi-engine ecosystem. Best-fit scenarios — Large-scale analytical lakes, cross-engine sharing, long-term schema evolution. Boundary — High-frequency primary-key updates and full changelog not core design focus.

Hudi : Standout capability — Upsert, indexing, COW/MOR, incremental queries, full table services. Best-fit scenarios — Update-intensive data lakes, CDC, incremental ETL. Boundary — Many tuning knobs for indexing, compaction, clustering; heavier ops burden.

Delta Lake : Standout capability — Transaction log, MERGE, Deletion Vector, Change Data Feed, mature Spark experience. Best-fit scenarios — Spark/Databricks ecosystems, ETL with streaming-batch unification. Boundary — Cross-engine feature completeness still needs per-engine verification.

Kafka : Standout capability — Low-latency ordered log, pub/sub, replay, mature ecosystem. Best-fit scenarios — Event bus, system decoupling, real-time transport. Boundary — Not an OLAP table store; historical analysis and primary-key snapshots need extra systems.

Fluss : Standout capability — Log table, primary-key table, sub-second streaming read/write, point lookup, lake tiering. Best-fit scenarios — Real-time analytics hot layer, feature store, state-intensive stream compute. Boundary — New project; ecosystem and long-term production experience still accumulating.

If Flink streaming compute is core and you want CDC, table updates, and historical analysis unified on object storage, Paimon fits well. If cross-engine open standards matter most, Iceberg is more natural. Existing Spark or Hudi investments make migration cost a key variable. Selection should weigh primary write engine, update ratio, query latency, object-store cost, and team ops capability—not just feature lists.

05 Why Fluss Emerges as a New Streaming Storage

Paimon already supports incremental reads/writes like a stream. Yet "can be consumed as a stream" differs from "built for sub-second streaming reads/writes." Paimon's new data becomes visible after snapshot commit; end-to-end latency is affected by Flink checkpoints, commits, and multi-layer propagation. Its strengths are openness, low cost, history retention, and batch analytics efficiency—not extreme local-disk hot-data serving.

Fluss positions differently: it combines replayable logs, Primary Key Tables, columnar streaming storage, and local hot data in one system, with a Tiering Service that continuously sinks historical data into open lake formats like Paimon, Iceberg, or Lance.

In a Fluss+Paimon combo:

Fluss holds latest, hottest data for sub-second streaming reads/writes, primary-key point lookups, real-time features.

Paimon holds full history for low-cost storage, large scans, time travel, multi-engine analytics.

Union Read reads Paimon snapshot first, then continues from aligned Fluss offset, stitching history and real-time into one logical table.

Bucket alignment between Fluss and Paimon reduces shuffle during tiering.

This shifts real-time warehouse architecture from "message queue + lake table + online DB" stitching to "hot streaming storage + open lake storage" layering: hot layer chases low latency, cold layer chases low cost and openness, both sharing table semantics and unified access. It doesn't replace Kafka or Paimon—Kafka remains the mature event bus; Paimon the open historical layer; Fluss adds a storage layer purpose-built for real-time analytics and state access between them.

Conclusion: Paimon's Value Is Making Real-Time Data Finally One Table

Real-time warehouse complexity often lies not in a single SQL but in the same data existing simultaneously in message queues, lakes, warehouses, caches, and serving DBs. Longer chains mean more state, harder recovery, backfill, and reconciliation.

Paimon's core capability organizes latest state, historical snapshots, and incremental changes around one table. It uses Snapshot and Manifest for versioning, LSM and Deletion Vector for updates, Changelog for downstream connectivity, and object storage for scale and cost advantages.

If first-gen real-time warehouses solved "can data be computed in real time", Paimon addresses "can computed data be stored in a unified, reliable, replayable way". Fluss+Paimon tackles the next question: can real-time hot data and low-cost historical data truly share one table semantics.

Choose Paimon not because it has many features, but because your architecture genuinely needs to converge streams, tables, and history into a single storage fact layer.