Understanding Apache Paimon Table Modes and Their Use Cases

Apache Paimon is a data‑lake solution that supports a variety of scenarios; this article introduces the different table modes it offers and the contexts in which each should be used.

Primary Key Table : The core of Paimon as a streaming data lake, it can ingest CDC or Flink streaming changelogs containing INSERT, UPDATE, and DELETE records. It supports Hive‑style partitioning, and data is distributed into buckets; each bucket holds an independent LSM tree and its changelog files. Bucket count limits parallelism, and a bucket size of roughly 200 MB‑1 GB is recommended.

Compaction : By default performed automatically in the Flink sink. To avoid write blocking, enable fully asynchronous compaction (e.g., set num-sorted-run.stop-trigger = 2147483647, sort-spill-threshold = 10, changelog-producer.lookup-wait = false). For multiple jobs writing the same table, configure write-only = true on writers and run a dedicated compaction job.

Fixed‑Bucket Primary Key Table : Define a fixed bucket count via the 'bucket' property (e.g.,

CREATE TABLE MyTable ( user_id BIGINT, item_id BIGINT, behavior STRING, PRIMARY KEY (user_id) NOT ENFORCED ) WITH ( 'bucket' = '5' );

). The bucket number influences concurrency and performance; the primary key must include all partition fields because cross‑partition updates are not supported.

Dynamic‑Bucket Primary Key Table : Set 'bucket' = '-1' so Paimon automatically manages bucket numbers. Two key parameters control its behavior: 'dynamic-bucket.target-row-num' (target rows per bucket) and 'dynamic-bucket.initial-buckets' (initial bucket count). Dynamic buckets cannot be written by multiple jobs concurrently; use a single Flink sink with UNION ALL if needed.

Normal Dynamic Bucket : Suitable when updates stay within partitions. It uses a hash index to map primary keys to buckets, consuming memory proportional to the number of entries (e.g., ~1 GB for 100 M entries). Example DDL:

CREATE TABLE MyTable ( user_id BIGINT, item_id BIGINT, behavior STRING, dt STRING, PRIMARY KEY (dt, user_id) NOT ENFORCED ) WITH ( 'bucket' = '-1' );

.

Cross‑Partition Update Dynamic Bucket : When the primary key does not contain all partition fields, this mode maintains a mapping of primary key to partition and bucket on local disk, initializing indexes quickly and using Flink managed memory as RocksDB block cache. Index TTL can be tuned via 'cross-partition-upsert.index-ttl' to reduce memory usage.

Append Table : Tables without a primary key; only INSERT operations are allowed. Compaction merely merges small files. Two modes exist: Scalable and Queue.

Append Scalable Table : Define 'bucket' = '-1' without a primary key; the table behaves like an enhanced Hive table without bucket constraints, supporting batch‑batch, batch‑stream, stream‑batch, and stream‑stream workloads. Use cases include replacing Hive tables with ACID guarantees, high‑throughput streaming, and fast point‑range queries using Z‑order.

Append Queue Table : Define a bucket and a 'bucket-key' so that records within the same bucket are strictly ordered, forming a queue‑like structure. Example DDL:

CREATE TABLE MyTable ( user_id BIGINT, item_id BIGINT, behavior STRING, dt STRING ) WITH ( 'bucket' = '10', 'bucket-key' = 'user_id' );

. This mode enables ordered streaming reads; currently it does not support the Flink‑Kafka default mode but is on the roadmap.

Both Append modes perform compaction to merge small files; the Queue mode may generate more small files, which can be reduced by lowering 'compaction.max.file-num'. Additionally, the Queue mode supports defining watermarks and aligns with the latest watermark strategies.