Taming Small Files in Paimon: Proven Tuning Strategies for Better Performance

Hello again! Today we share practical ways to manage small files in Paimon.

In Paimon's streaming data lake architecture, small files strain the underlying distributed file system (e.g., HDFS), overload NameNode metadata, and hurt query efficiency.

1. Parameter Tuning

1.1 Flink Job Parameter Optimization

1.1.1 Adjust Checkpoint Interval

The checkpoint interval is a key factor; each checkpoint forces Paimon's writer to flush its in‑memory WriteBuffer to the file system, creating new files.

Increasing the interval reduces file creation but adds data visibility latency.

1.1.2 Set Maximum Concurrent Checkpoints

In distributed environments, checkpoint “tails” can appear. Setting execution.checkpointing.max-concurrent-checkpoints limits the number of simultaneous checkpoints, alleviating the tail problem.

1.1.3 Adjust Sink Parallelism

The sink parallelism (sink.parallelism) directly influences small‑file generation and write performance. It should match the number of buckets so each subtask writes to its own bucket, avoiding shuffle. Higher parallelism improves throughput but may increase small‑file count and resource consumption.

1.2 Paimon Table Parameter Optimization

target-file-size

write-buffer-size, write-buffer-spillable

bucket

1.2.1 Set Target File Size

The target-file-size defines the desired size of files produced by compaction. Larger target sizes reduce the number of small files and improve query performance.

1.2.2 Adjust Write Buffer Size and Spill Strategy

The writer buffers data in memory before flushing. The buffer size is controlled by write-buffer-size (default 128 MB). Increasing it allows more data to be accumulated, producing larger L0 files and fewer small files.

When write-buffer-spillable is set to true, a full buffer spills to a temporary local file before flushing, which is recommended for production.

1.2.3 Optimize Bucket Number

Data is physically organized by partitions and buckets. Each bucket maps to an independent LSM‑Tree and write channel. The bucket count determines write concurrency and file organization; a practical rule is to keep each bucket around 1 GB of data.

1.3 Asynchronous Small‑File Merging

Enable asynchronous compaction in production to merge small files in the background.

2. Operations Monitoring

Key monitoring metrics for Paimon writes include the following (illustrated below):

That concludes our sharing on small‑file governance in Paimon.