Apache Paimon Internals: LSM Write Path, Compaction Scheduling & Changelog Production

1. Problem Background and Technical Evolution

The previous article established that streaming-batch unification is about storage unification, not language unification. The component that truly bears the burden of being continuously written by Flink, batch-recomputed by Spark, and read by analytical engines like StarRocks is the data lake table format. First-generation lake table formats (Iceberg, Hudi) centered on immutable batch files and snapshot commits. When businesses continuously write CDC primary key tables, real-time dimension tables, and streaming results into lake tables, the contradiction between real-time update writes and batch query read amplification surfaced.

Apache Paimon essentially introduces the LSM-Tree storage paradigm—designed for high-frequency updates and read-time merging—into open lake table storage. It no longer treats primary key tables as independent batch files but organizes them via Buckets, Sorted Runs, and Compaction into a continuously merging LSM tree. This design choice makes its write path, compaction scheduling, changelog production, and failure recovery distinctly different from traditional lake storage built around immutable data files.

A common misconception must be corrected: Paimon using the LSM model does not mean its physical data files are RocksDB-style SST files. Paimon's default data file format can be Parquet, and primary key data files also store Row Kind, Sequence Number, and other information for version merging. Therefore, LSM describes Paimon's data organization, write, and merge model, not a direct equivalence to RocksDB's file format.

Why source-code-level analysis today? Because issues like compaction backlog, small file explosion, and changelog latency are no longer abstract metrics but map to specific components, stages, and parameters. Without understanding the kernel, troubleshooting reduces to restarts and luck.

2. What You Will Gain

A complete source-code trace of a record entering a Paimon primary key table through Sink, becoming data files, forming Sorted Runs, and committing snapshots, enabling a mental model of key classes and state transitions.

Clear explanation of the LSM write path (memory buffer → sort → data file → Sorted Run) and why compaction is a trade-off of write amplification for read optimization.

Understanding of the four changelog production modes (none, input, lookup, full-compaction), their costs, applicable scenarios, and why changelog is key to streaming-batch unification.

Mastery of how primary key tables, append tables, Buckets, snapshot commits (two-phase commit) work at the kernel level.

A source-code-level troubleshooting path for compaction backlog, small files, changelog latency, and snapshot bloat.

Ability to articulate why Paimon suits real-time updates as a mechanism, not just a speed comparison.

3. Core Concepts and Architecture Positioning

Paimon tables mainly split into Primary Key Tables and Append Tables . Primary key tables use LSM-Tree, supporting updates, deletes, and version merging by primary key; append tables target append-only data without primary key update maintenance. Both share Snapshot, Manifest, and other lake metadata systems, but write and query paths differ significantly—the prerequisite for understanding all problems.

Logically, a Paimon primary key table is: Table → Partition → Bucket → Sorted Run → Data Files . Bucket is the basic LSM Tree organizational unit; each Bucket owns its LSM Tree. Under fixed Bucket mode, data hashes to fixed Buckets by Bucket Key; current Paimon also supports Dynamic Bucket and Postpone Bucket, so it cannot be simply summarized as "fixed by hash at table creation, immutable at runtime." Writes route to corresponding Buckets per distribution strategy; compaction and reads revolve around the Bucket's internal LSM Tree.

In process roles, Paimon as a Flink Sink runs on TaskManager write operators. Its kernel roughly divides into three layers: Write Layer (receive records, enter write buffer, sort and generate data files), Compaction Layer (background merge of multiple Sorted Runs to reduce subsequent read-time merge work), Commit & Metadata Layer (atomically publish a write via Snapshot and Manifest). It does not manage Flink job resource scheduling but combines with Flink's Checkpoint mechanism to complete streaming write commits.

4. Deep Dive into Underlying Principles

4.1 Write Path: How Memory Buffer Becomes Sorted Run

A record entering a primary key table does not directly modify existing data files but first enters the write buffer. The write operator maintains in-memory structures for sorting and file writing; new records are organized by primary key and version info. When flush conditions are met, data is sorted and written as new data files, forming new Level 0 Sorted Runs.

A frequent source-code misunderstanding must be corrected: Paimon primary key table physical data files cannot be directly called RocksDB SST files. Paimon Data Files default to Parquet format and store Row Kind, Sequence Number, etc., for subsequent merges. The accurate source-code chain is:

Record → Write Buffer → Sort → Data File → Level 0 Sorted Run

Not:

Record → RocksDB MemTable → SST

Data files being internally ordered is crucial : it enables efficient multi-way merge during subsequent compaction, rather than unordered reshuffling of all data each time.

Why sort in memory before dropping to L0? Because LSM benefits rely on Sorted Runs being ordered. If on-disk files were completely unordered, subsequent compaction and queries would pay higher costs. The cost: writes need buffering, sorting, and flush; in Flink streaming writes, data commit visibility is also affected by Checkpoints—thus Checkpoint interval directly participates in the latency trade-off from write to visibility , but cannot be simply understood as "only Checkpoint triggers file flush."

4.2 Tiered LSM: Why Read Must Merge, Background Must Compact

Each Bucket's data is organized by Sorted Runs. Level 0 typically contains newer data files; different Runs may overlap. As compaction proceeds, multiple Runs merge into new Runs, reducing the number of data files and versions queries must process.

When querying a primary key, the latest data may be distributed across multiple Runs, requiring merge-on-read —read candidate data, then derive final logical result by Primary Key, Sequence Number, Row Kind, and configured Merge Engine.

Merge-on-read is flexible but costs read amplification : more Sorted Runs mean more files to open, read, and merge for queries, impacting point lookups and scans. This is the fundamental reason for compaction: a background asynchronous task that merges multiple Sorted Runs per strategy, incidentally handling "multi-version merge for same primary key" and "delete record processing" per Merge Engine, reducing version count for subsequent reads.

Note : compaction does not guarantee "never blocks writes." Normally Paimon compacts in background, but when Sorted Run count hits stop threshold, or certain modes require compaction completion before commit, writes may face backpressure or wait; e.g., lookup Changelog Producer defaults to waiting for corresponding Lookup Compaction.

Key design philosophy : Compaction trades write amplification + background overhead for read performance. Write path defers heavy merge work to background, gaining better sustained write throughput; the cost is compaction must keep up with write speed, else Sorted Runs accumulate, increasing read amplification, memory pressure, and file count. The previous article's "compaction is primary key table ops core" stems from this kernel reason.

4.3 Changelog: Why LSM Table Still Produces Change Stream

A natural question: LSM files can derive "latest result" via Merge Engine, why does Paimon produce Changelog? Because downstream streaming compute (another Flink job, dimension table join, real-time features) needs not "current full latest state" but "what changed" (+I/-U/+U/-D Changelog types). Directly reading final table state does not inherently contain before-image of each update.

Paimon solves this via changelog-producer parameter, with four mainstream modes:

none : no extra full changelog files, but streaming reads can still read incremental changes; downstream can work if it handles Upsert or Flink normalizes. It is not "batch snapshot read only" nor full CDC audit log.

input : writes upstream input changelog into independent changelog files for streaming reads; requires upstream to already provide needed change semantics (e.g., CDC records with before/after); Paimon does not fill missing before images.

lookup : reads existing state via Lookup Compaction, applies new data to generate corresponding changelog. Suits scenarios where upstream lacks before image but downstream needs full change semantics; cost is extra lookup, compaction, cache, and I/O overhead.

full-compaction : compares table state before and after Full Compaction to generate changelog. Describes changes between two Full Compaction states; intermediate multiple updates may be folded, not a full replica of every source event.

These four modes are not a simple "zero-cost vs semantic completeness" continuum, but different consumer contracts with compute/storage costs . Selection principle: downstream handles Upsert → none; upstream provides full changelog → input; no before image but downstream needs full changes → lookup; downstream accepts periodic latency, wants state diff via Full Compaction → full-compaction . This is the interview point that best distinguishes "actually used it."

4.4 Snapshot Commit: How Two-Phase Commit Guarantees Exactly-Once

A Paimon write is not "visible once files written" but publishes committed table state via Snapshot . Each Checkpoint cycle, write operator prepares pending data files and commit info; after Flink Checkpoint succeeds, Paimon Writer's commit chain continues Commit, new Snapshot published, visible externally.

Three concepts must be distinguished:

Flink Checkpoint ≠ Paimon Snapshot.

Checkpoint is Flink's state consistency mechanism; Snapshot is Paimon's table state metadata. They connect via Paimon Sink's commit protocol. Official docs clarify: Streaming Writer commits data as Checkpoint completes; readers query committed Snapshots.

This mechanism brings three key properties : first, read-write isolation —queries read committed table state, never see unpublished data files; second, under Flink Source, Checkpoint, Sink end-to-end semantic requirements, exactly-once data commit semantics achievable; third, time travel —retained Snapshots support historical state queries. Cost: snapshots and referenced data files cannot delete while still needed; longer Snapshot retention increases storage and metadata management cost.

5. Core Chains and Key Flows

Stringing the four threads into the most critical path: one Flink streaming write into Paimon primary key table producing changelog .

Write operator reads upsert record from Kafka/Fluss → routes to target Bucket per Bucket distribution strategy → enters that Bucket's active write buffer, organized by primary key and version info → if lookup mode configured, Lookup Compaction reads existing state and generates changelog → upon flush condition, data sorts and writes as new data file, forming L0 Sorted Run → background compaction merges multiple Sorted Runs per strategy, incidentally merging multi-versions and handling deletes per Merge Engine → Flink Checkpoint completes, Paimon Commit publishes corresponding Snapshot → downstream streaming read job continues reading incremental changes from committed Snapshot.

Another critical path is failure recovery : job fails and restarts, Flink recovers state and consumption offsets from latest successful Checkpoint, Paimon continues writing from successfully committed table state; files not yet in valid Snapshot do not become part of normal table state, handled by Paimon's subsequent file cleanup mechanism, not simply "task fails immediate delete." Committed Snapshots remain unchanged; downstream won't read a half-committed state due to one failure. Understanding this chain explains why Paimon can collaborate with Flink Checkpoint on lake tables for end-to-end consistency.

6. Real Business Scenarios and Engineering Practice

Take e-commerce "real-time user wide table": upstream writes user basic attributes, orders, behavior three CDC streams into Kafka/Fluss; a Flink job aggregates by user_id into Paimon primary key table (partial column updates); downstream real-time recommendation and risk control directly read this table for dimension table join.

Key engineering decisions:

First, Bucket mode and count : plan by write parallelism, data skew, update frequency, file size. Fixed Bucket needs upfront planning, but current Paimon supports Dynamic Bucket and Postpone Bucket, so "Bucket immutable after creation" no longer holds; if fixed Bucket layout adjustment needed, official Rescale Bucket process handles it.

Second, changelog-producer selection : if upstream already provides full CDC changelog, prioritize input; if upstream lacks before image but downstream needs full -U/+U, consider lookup; if downstream accepts periodic state changes, evaluate full-compaction.

Third, snapshot retention : production sets by "how far back to trace, streaming job recovery window, batch query duration", e.g., retain days for data traceback and recovery; infinite retention not recommended.

Table creation revolves around these points with key parameters (specific values per version docs): specify primary key, Bucket mode and count, Bucket Key, merge engine, Sequence Field, changelog mode, snapshot retention policy, compaction params. No large SQL dumps because success hinges on selection judgment, not parameters themselves.

7. Performance, Cost, and Technology Selection

Paimon primary key table performance characteristics all derive from LSM write, merge-on-read, background compaction .

Write latency affected by write buffer, flush, checkpoint, commit, compaction—cannot simply blame checkpoint interval; low-latency scenarios must holistically trade off checkpoint interval, file generation frequency, commit overhead.

Read performance depends on Sorted Run count, file layout, merge-on-read, and whether compaction keeps up; compaction backlog can significantly degrade read latency.

Storage cost determined by data files, changelog files, compaction rewrites, snapshot retention.

When to use primary key table: high-frequency CDC primary key updates, real-time dimension tables, streaming results needing primary key state maintenance.

When not to use: pure append massive logs—prefer Append Table to avoid unnecessary primary key merge semantics. In division with Iceberg/Hudi: for Flink real-time updates, primary key state maintenance, streaming incremental consumption as main, Paimon's LSM model fits better; pure offline large-scale batch analysis, rare changes—evaluate Iceberg/Hudi per query engine, ecosystem, data management needs. All throughput, compression ratio, target file size numbers are environment-dependent experience starting points; must benchmark on own hardware, payload, concurrency—cannot copy blindly.

8. Production Issues and Troubleshooting

Failure 1: Compaction backlog, small file explosion. Phenomenon: Bucket Sorted Run/data file count continuously rises, query latency increases. Follow "phenomenon → metrics → principle → locate → fix → prevent": first check each Bucket's Sorted Run count and compaction progress; principle: write speed exceeds background merge speed; common root causes: insufficient compaction resources, compaction task errors, unreasonable Sorted Run trigger thresholds, or excessive lookup/full compaction work. Fix: add compaction resources per business, use independent compaction job, or run targeted full compaction during low traffic; prevent: reverse-engineer compaction compute from peak write speed, avoid long-term resource contention between compaction and business writes.

Failure 2: Changelog latency or semantic anomaly. Phenomenon: downstream streaming job sees missing -U/+U, high latency, or unexpected semantics. Principle: directly tied to changelog-producer selection: input mode relies on upstream full changelog; lookup mode relies on lookup compaction, async compaction may further delay changelog availability; full-compaction mode only produces state diff after corresponding full compaction. Locate: confirm mode used, whether upstream records carry before image, whether downstream needs Upsert or full changelog; fix: switch mode per consumer contract, not simply by reducing upstream disorder.

Failure 3: Snapshot bloat causing storage rise. Phenomenon: write volume stable but storage keeps growing. Principle: snapshot retention too long, changelog files increase, compaction rewrites, historical snapshots still reference data files. Fix: adjust snapshot retention per business, check Tag/Branch long-term references, reclaim unneeded data via Paimon's file lifecycle management and orphan file cleanup. Critical boundary : never manually delete Paimon data files in object storage/HDFS—that breaks Snapshot/Manifest-to-physical-file references, may directly cause table read failure—a low-level accident Paimon ops must strictly avoid.

9. Advanced Interview and Architecture Thinking

Q1: Why is Paimon suitable for real-time updates? Explain as mechanism. Key points: Paimon primary key table adopts LSM storage model; writes enter buffer and form new Sorted Runs, no need to immediately rewrite entire historical data files per update; background compaction gradually merges multiple versions. Thus it decouples high-frequency update path from historical reorganization path, better fitting CDC and continuous update workloads. Don't simply answer "Paimon definitely faster than Iceberg"—final performance depends on specific update ratio, file layout, query pattern, resource config.

Q2: How to choose among four changelog-producer modes? Must articulate "consumer semantics vs extra cost" trade-off: downstream handles Upsert → none; upstream provides full changelog → input; need before image from existing state → lookup; accept periodic state changes, want changelog via full compaction → full-compaction.

Q3: Why does compaction backlog? How to stress-test? Essence: race between write speed and background merge speed; stress test: replicate peak write proportionally, monitor Sorted Run count, compaction backlog, query latency over time—if not converging, current compaction capacity cannot cover write pressure; further check CPU, IO, network, file size, compaction strategy.

Q4: Job fails and restarts, why downstream doesn't read duplicate or jump data? Key points: Flink Checkpoint and Paimon Commit collaborate; only successfully committed and published Snapshots become normal table state externally; job recovers from latest successful Checkpoint state and offsets; uncommitted data files don't directly become new table state. Note: "end-to-end exactly-once" not guaranteed by Paimon alone, but jointly by Source, Flink Checkpoint, Sink Commit, and downstream consumption semantics.

10. Summary, Knowledge Distillation, and Next Episode

This article grounded the four keywords from previous episode into Paimon code:

Storage unification : a primary key table maintains continuously changing logical state via LSM, while enabling batch reads and incremental consumption via Snapshot.

Compaction : core mechanism where background merges multiple Sorted Runs, reduces read-time merge work, trades write amplification for read performance.

Late-arriving updates : embodied via Sequence, Merge Engine, and multi-version merging.

Two-phase commit : via Flink Checkpoint and Paimon Commit collaboration, publishing completed data as new Snapshot.

Per chapter 2: we traced write buffer → data file → Sorted Run → commit source-code chain, clarified four changelog mode trade-offs, mastered source-code-level troubleshooting for Bucket, compaction, snapshot, changelog three failure categories, and articulated "why Paimon suits real-time updates" as mechanism.

One-sentence conclusion: Paimon's kernel is using LSM's continuous write and background merge to turn lake table from pure batch file collection into sustainably updatable, stream-readable, version-managed state storage—its strengths and ops pain points both stem from this model choice.

Next episode (Part 11) dives into Flink tightly bound with Paimon commit: Checkpoint and State Backend Source Code Deep Dive —how Barrier aligns and propagates, why Unaligned Checkpoint alleviates snapshot timeout under backpressure, how RocksDB/ForSt state backends and incremental snapshots work, full flow of Savepoint and failure recovery. Today's "Checkpoint participates in write commit two-phase commit" finds its true source in Flink side next time.