Key Considerations for Using Paimon Primary Key Tables

Today we discuss some usage considerations for Paimon primary key tables.

1. Primary Key Table

A primary key table is a type of Paimon table that allows users to insert, update, or delete records, essentially enabling a unique primary key that can be overwritten.

Bucket Selection

Regardless of whether the table is partitioned, a bucket is the smallest read/write unit, containing an LSM tree and its change log file.

Users can specify a bucket-key option to define the bucketing column; if omitted, data is distributed based on the hash of the entire record (in recent versions, creating an Append Queue table requires a bucket-key). When a primary key is defined, hashing is based on the primary key.

The number of buckets limits the maximum parallelism. The recommended bucket size is 200 MB–1 GB (2 GB is also workable), but the total number of buckets should not be excessive.

Paimon also supports bucket rescaling, providing a Rescale Bucket feature to redistribute historical data.

Cross‑Partition Update Issue

If a primary‑key table is partitioned (e.g., daily partitions) but the primary key does not include the partition key, cross‑partition updates can occur, and behavior varies by merge engine:

Deduplicate: delete old partition data and insert new data into the new partition.

PartialUpdate & Aggregation: insert new data into the old partition.

FirstRow: ignore new data when an old value exists.

Large data volumes can cause noticeable performance degradation.

Therefore, it is strongly recommended to avoid partitions when using primary‑key tables and to combine them with record‑level expiration capabilities.

The core configuration consists of two settings:

'record-level.expire-time': retention time in seconds 'record-level.time-field': expiration field of type INT

File Compaction

Compaction is a resource‑intensive process that consumes CPU and disk I/O; overly frequent compaction can slow write throughput.

Paimon provides a default compaction strategy that works for most scenarios, but if compaction causes lag, two alternatives are available:

Switch from synchronous to asynchronous compaction by enabling two parameters:

num-sorted-run.stop-trigger = 2147483647<br/>sort-spill-threshold = 10

Launch a dedicated compaction task using the Dedicated Compaction strategy (similar to Hudi's offline compaction).

This approach allows a separate job to handle file merging.

Example command:

<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-0.8.2.jar \
    compact \
    --warehouse s3:///path/to/warehouse \
    --database test_db \
    --table test_table \
    --partition dt=20221126,hh=08 \
    --partition dt=20221127,hh=09 \
    --table_conf sink.parallelism=10 \
    --catalog_conf s3.endpoint=https://****.com \
    --catalog_conf s3.access-key=***** \
    --catalog_conf s3.secret-key=*****

Below are several promotional links and calls to action that were included in the original article.

300万字！全网最全大数据学习面试社区等你来！

If this article helped you, please remember to "watch", "like", and "bookmark".