Avoid These 6 Common Paimon Data Loss Pitfalls in Flink and Spark

Hello everyone, today we briefly discuss several scenarios that can cause data loss when writing to Paimon.

Paimon is one of the most widely used frameworks in the data lake domain. The following operations should be avoided to prevent data loss during data ingestion into Paimon:

1. Checkpoint failure and forced restart leading to uncommitted data loss.

Insufficient resource allocation for Flink writing to Paimon can cause CPU to reach 100%, checkpoint timeout, and a forced task restart, resulting in loss of data after the latest successful checkpoint.

Root cause: Flink only commits to Paimon after a checkpoint succeeds; a failed checkpoint followed by a forced restart discards the in‑memory buffer.

2. In partial‑update mode, delete messages are ignored, so upstream deletions are not reflected downstream.

Root cause: The partial-update mode is designed to update specific fields without handling row deletions. To detect deletions, add a dedicated delete‑flag column (e.g., is_deleted) and configure Paimon to recognize it.

3. Misconfiguration of sequence.field causing old data to overwrite new data.

Root cause: Selecting an incorrect ordering field leads to out‑of‑order data. Use a monotonic field as sequence.field and, when merging multiple streams, assign distinct sequence fields to each stream instead of sharing one.

4. Snapshot expiration too fast; failover beyond 2 hours makes the job unable to find files.

Root cause: The default snapshot.time-retained is 1 hour, causing snapshots to be cleaned up. For streaming tables, always set a consumer ID and keep the retention time at least as long as the maximum downtime (recommended starting from 24 hours).

5. Two jobs concurrently writing to the same bucket cause the second job to fail continuously.

Root cause: Paimon guarantees only snapshot isolation for concurrent writes to the same bucket; conflicts trigger infinite retries until timeout, appearing as data loss. Ensure only one job writes to a table (set write-only=true) or, if dual writes are needed, use different bucket fields or enable dynamic-bucket=true.

6. Spark writing to an older Paimon version with insufficient memory leads to retry failures where later attempts overwrite earlier ones.

Root cause: Before Paimon 0.9, Spark writes are non‑atomic; on retry, a new file is written before the previous one is committed, causing the new file to overwrite the old one. Upgrade Paimon to a newer version promptly.