Performance Optimization of Apache Paimon in Dolphin OLAP Engine

The article introduces Apache Paimon, a lake‑format storage engine incubated from the Flink community, and its integration with Alibaba’s Dolphin multi‑model AI‑enhanced database for OLAP workloads.

It explains why Paimon is chosen as the storage protocol: support for streaming writes, primary‑key updates, compute‑storage separation, open source, and active community.

Dolphin uses Paimon as an external table format, enabling batch‑stream convergence and allowing most queries to achieve millisecond‑level latency with up to 10k QPS.

Performance Test Report

Cold‑query benchmarks show Dolphin‑Paimon outperforming StarRocks and Hologres by 2× to over 50× in most scenarios. High‑QPS tests on a 100‑node cluster (16 cores, 64 GB, Pangu HDD) demonstrate the ability to handle large‑scale point‑lookup workloads.

Optimization Paths

1. Enable ORC push‑down – Activating ORC filter push‑down and selected‑row reading yields dozens of times performance gains. The relevant code changes are:

package org.apache.paimon.format.orc;
public class OrcReaderFactory implements FormatReaderFactory {
//...
private static RecordReader createRecordReader(
org.apache.hadoop.conf.Configuration conf,
TypeDescription schema,
List<OrcFilters.Predicate> conjunctPredicates,
FileIO fileIO,
org.apache.paimon.fs.Path path,
long splitStart,
long splitLength){
// ...
if (!conjunctPredicates.isEmpty()) {
// TODO fix it , if open this option,future deletion vectors would not work,
//  cased by getRowNumber would be changed .
options.useSelected(OrcConf.READER_USE_SELECTED.getBoolean(conf));
options.allowSARGToFilter(OrcConf.ALLOW_SARG_TO_FILTER.getBoolean(conf));
}
// ...
}
//...
}

Configuration parameters to enable the push‑down:

'orc.filter.use.selected' = 'true',
'orc.sarg.to.filter' = 'true'
# GitHub version parameters
'orc.reader.filter.use.selected'='true',
'orc.reader.sarg.to.filter'='true'

2. Convert manifest to ORC format – The default Avro manifest requires full in‑memory loading. Switching to ORC reduces metadata size and I/O. Example manifest entry:

{
"_VERSION": 2,
"_KIND": 0,
"_PARTITION": [...],
"_BUCKET": 47,
"_TOTAL_BUCKETS": 100,
"_FILE": {
"_FILE_NAME": "data-...-0.orc",
"_FILE_SIZE": 3088,
"_ROW_COUNT": 1,
...
}
}

Reading logic based on bucket, partition, and filter can cause millions of loop iterations. By applying bucket push‑down (PR #4497) the metadata read volume drops by 10,000×.

3. Reduce I/O operations – Cache schema and snapshot files (few KB) to avoid frequent reads, and enable manifest caching with options such as:

Options options = new Options();
options.set(WAREHOUSE, uri.toString());
options.set("cache-enabled","true");
options.set("cache.expiration-interval","1 min");
options.set("cache.manifest.small-file-memory","10m");
options.set("cache.manifest.small-file-threshold","3mb");
CatalogContext context = CatalogContext.create(options,...);

Be cautious of JVM GC impact when caching.

4. Increase concurrency – Execution plans generate orthogonal sub‑plans that can be read in parallel, improving throughput by over 10× for many ORC files.

5. Avoid redundant encoding/decoding – Re‑encoding of strings during setString, length calculation, and write operations was eliminated by introducing a static byte buffer and a custom SpeedGPDWritable implementation.

These low‑level optimizations together bring Dolphin‑Paimon to millisecond‑level latency with QPS >10k, and the system now serves thousands of tables at PB scale.

Business Impact and Future Work

Current production metrics: >1000 QPS, p99 latency ~100 ms, >1000 tables, PB‑scale data. Future plans include a second‑level query solution for massive log aggregation and a 100k QPS, <5 ms latency implementation. Integration with ClickHouse is also in progress.