StarRocks Integrates Paimon C++ for 1.4–3.4x Faster Lake Queries via Native Execution

In real-time lakehouse architectures, Apache Paimon manages continuously changing lake data while StarRocks provides reporting, interactive analysis, and cross-source joins via Paimon Catalog. However, querying Paimon tables — especially primary-key tables — requires understanding snapshot isolation, version merging, and delete-vector semantics, tasks previously handled by StarRocks calling the Paimon Java SDK through JNI.

As data volume, table width, and concurrency grow, the JNI boundary introduces measurable overhead: repeated Java↔C++ transitions, data serialization/deserialization, and opaque resource accounting make it hard to pinpoint whether latency stems from I/O, merge logic, or downstream computation.

Paimon C++ (paimon-cpp) as a Native Reader

Paimon C++ is a C++ reimplementation of the Paimon format reader. It runs inside the StarRocks BE process, requires no separate deployment, and preserves existing SQL semantics. Unlike the built-in file reader that only scans Parquet/ORC, paimon-cpp executes the full Paimon merge logic (primary-key version reconciliation, delete-vector application) natively, allowing more queries to stay on StarRocks' vectorized C++ execution pipeline.

Three Readers and Their Roles

StarRocks File Reader — direct Parquet/ORC scan; used when a read task can be safely reduced to file scanning (common for append-only tables, but also possible for some primary-key tables).

Paimon C++ Reader — native handling of Paimon table semantics (primary-key merge, delete vectors); covers tasks that cannot be simple file scans.

JNI Reader — falls back to the Paimon Java SDK for system tables or compatibility diagnostics.

Reader Selection via paimon_reader_mode

AUTO

(default): prefers file reader for scannable tasks; remaining tasks use JNI. Does not auto-enable paimon-cpp. NATIVE: routes supported tasks to Paimon C++ Reader; system tables may still use JNI. JNI: forces Java SDK path for compatibility checks or side-by-side benchmarking.

Selection happens per scan task; a single SQL with multiple scans may use different readers. To evaluate native performance, explicitly set NATIVE and verify the actual path via Query Profile.

Enabling Native Reads

Create a Paimon Catalog (example uses Hive Metastore + HDFS via DLF REST):

CREATE EXTERNAL CATALOG paimon_lake
PROPERTIES (
  "type" = "paimon",
  "paimon.catalog.type" = "rest",
  "uri" = "https://cn-hangzhou-vpc.dlf.aliyuncs.com",
  "paimon.catalog.warehouse" = "<dlf_catalog_name>",
  "token.provider" = "dlf"
);

Confirm the session variable exists: SHOW VARIABLES LIKE 'paimon_reader_mode'; Disable the higher-priority JNI override and switch to native mode with profiling:

SET paimon_force_jni_reader = false;
SET paimon_reader_mode = 'NATIVE';
SET enable_profile = true;

Run existing SQL unchanged, e.g.:

SELECT order_date, SUM(amount) AS paid_amount
FROM paimon_lake.sales.orders
WHERE order_date = '2026-09-01' AND order_status = 'PAID'
GROUP BY order_date;

Performance Evaluation Methodology

Compare NATIVE vs JNI on identical snapshots, SQL, and cluster resources. Test primary-key queries, wide-table aggregations, and common filter patterns. Control cache state (cold vs warm) separately to isolate reader gains from cache warm-up. Examine Query Profile for scan bytes, remote reads, cache hits, memory usage, and phase timings; if bottlenecks lie in join/aggregation, reader swap yields limited end-to-end improvement.

Benchmark Results (Single BE, 8-core CPU, 32 GB RAM)

Paimon C++ Reader delivered 1.4–3.4× speedup over JNI Reader:

Primary-key table count: 1,251 ms → 403 ms (~3.1×)

1,000-column wide-table full-width aggregation: 62.0 s → 18.5 s (~3.4×)

The advantage is most pronounced in primary-key merge and wide-table scenarios where JNI serialization and merge logic overhead dominated.

Future Roadmap

Expand native coverage : support more table types, data types, and schema-evolution cases while maintaining Paimon format compatibility.

Reduce query overhead further : optimize metadata preparation, remote I/O, merge algorithms, columnar decoding; enhance cache utilization and observability metrics.

Explore multi-modal and index acceleration : as Paimon adds BLOB, global indexes, vector, and full-text search, push filters and predicates down to lake indexes to cut data read, enabling unified SQL for analytics and search.

Related code is merged into StarRocks main branch and will ship in the next release. Subsequent articles will detail usage patterns per capability as they land.

Architecture diagram: Paimon C++ handles native reading; StarRocks handles query planning and parallel compute; data remains in Paimon.

Reader-selection flow: AUTO prefers file scan, NATIVE uses paimon-cpp for supported tasks, JNI forces Java SDK.