Apache Paimon in Practice: Replacing Hudi for Improved Write and Query Performance

In response to growing real‑time data demands, Tongcheng Travel migrated its lakehouse from Apache Hudi to Apache Paimon, achieving roughly three times faster writes and ten times faster queries.

1. Lakehouse Scenario Status and Problems

The company initially built a lakehouse on Hudi, using custom data‑integration pipelines to ingest binlog data, perform incremental reads, and generate near‑real‑time reports via Flink windowing. Over time, the Hudi‑based solution exhibited several issues: high write‑latency, slow point‑lookup performance, excessive resource consumption, and complex tuning parameters.

Key pain points included:

Parallelism and resource allocation for MOR tables were overly high.

Point‑lookup queries often took >20 seconds.

IO pressure on HDFS increased due to compaction and merge operations.

2. Encountering Apache Paimon

Apache Paimon (formerly Flink Table Store) is an incubating Apache project that provides a streaming data‑lake platform with high‑throughput ingestion, change‑data‑capture, and efficient real‑time analytics. It uses an LSM‑style storage layout, supports multiple cloud storage backends, and integrates with major query engines such as Flink, Spark, Hive, and Trino.

Key features highlighted:

Near‑real‑time updates and partial updates.

Local and global indexing, incremental stream reads, and full‑incremental hybrid reads.

Multi‑cloud storage and multi‑engine compatibility.

Built‑in CDC ingestion and schema evolution (in progress).

2.1 Performance Improvements with Paimon

Benchmarks showed that Paimon reduced container count and resource usage while maintaining write throughput. For a workload of 400 million rows across 800 partitions, Paimon completed the job in ~3 hours versus ~10 hours for Hudi MOR.

Point‑lookup latency dropped from 21 seconds (Hudi) to 2.7 seconds (Paimon) thanks to ordered storage and reduced scan volume.

parallelish.default : 2
execution.checkpointing.interval : 2 min
taskmanager.memory.process.size : 6g

3. Apache Paimon Application Practices

Integration pipelines were built to hide binlog details from users, providing one‑click full and incremental sync to Paimon tables. Large‑scale migrations from Hudi to Paimon used Flink batch jobs; tables with ~400 million rows were imported in under 20 minutes.

INSERT INTO paimon.ods.order_info
/*+ OPTIONS('sink.parallelism'='100','write-buffer-size'='1024m','sink.partition-shuffle'='true') */
SELECT * FROM hudi.ods.order_info/*+ OPTIONS('read.tasks'='100') */;

Partial‑update wide tables were created using the merge-engine='partial-update' setting, enabling efficient joins at storage level without retaining state in the compute engine.

CREATE TABLE IF NOT EXISTS paimon.dw.order_detail ( ... ) WITH ( 'bucket'='20', 'bucket-key'='order_id', 'sequence.field'='binlog_time', 'changelog-producer'='full-compaction', 'merge-engine'='partial-update', 'partial-update.ignore-delete'='true' );

Append‑only use cases (e.g., event logs) leveraged Paimon’s write-mode='append-only' to achieve real‑time ingestion and low‑cost reads.

CREATE TABLE IF NOT EXISTS paimon.ods.event_log ( ... ) PARTITIONED BY ( ... ) WITH ( 'bucket'='100', 'bucket-key'='uuid', 'snapshot.time-retained'='7 d', 'write-mode'='append-only' );

4. Issues Discovered and Solutions

Several problems were addressed:

Custom HiveCatalog implementation to support cross‑warehouse Paimon table locations in Spark.

Increasing Flink’s akka.framesize to avoid oversized payload errors during large partition/bucket reads.

Adding split‑batching logic to Flink static file store enumerator to mitigate Akka message size limits.

@Override
public Path getDataTableLocation(Identifier identifier) {
    try {
        Table table = client.getTable(identifier.getDatabaseName(), identifier.getObjectName());
        return new Path(table.getSd().getLocation());
    } catch (TException e) {
        throw new RuntimeException("Failed to get table location", e);
    }
}

2023-03-21 15:51:08,996 ERROR akka.remote.EndpointWriter [] - Transient association error (association remains live)
akka.remote.OversizedPayloadException: Discarding oversized payload sent to Actor[akka.tcp://flink@hadoop-0xx-xxx:29413/user/rpc/taskmanager_0#1719925448]: max allowed size 10485760 bytes, actual size of encoded class org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation was 1077637236 bytes.

5. Future Plans

Roadmap includes expanding Paimon’s ecosystem (analysis tools), building streaming data‑warehouses on top of Paimon, promoting internal adoption, and replacing certain message‑queue scenarios with Paimon‑backed pipelines.

References:

Apache Paimon website: https://paimon.apache.org/

Paimon GitHub: https://github.com/apache/incubator-paimon

Full‑incremental real‑time lakehouse with Flink Table Store: https://www.jianshu.com/p/ac2ba73367fe

Flink Table Store 0.3 best practices: https://juejin.cn/post/7207659644486959163