StarRocks Integrates Paimon C++ for Native Lake Query Execution

Background: Why Paimon C++?

In real-time lakehouse architectures, Apache Paimon manages continuously changing lake data while StarRocks provides reporting, interactive analysis, and cross-source joins via Paimon Catalog. Querying Paimon tables requires understanding table-format semantics such as snapshots, primary-key merges, and delete vectors — not just reading Parquet/ORC files. Previously, StarRocks handled these tasks by calling the Paimon Java SDK through JNI, which introduced cross-language call overhead, data conversion costs, and made it harder to attribute latency and memory usage to specific stages.

Solution: Native C++ Reading Path

Paimon C++ (paimon-cpp) is a C++ implementation of the Paimon format that lets native engines read Paimon tables directly. For StarRocks users it is an internal component; no separate deployment or SQL changes are required. By moving snapshot resolution, primary-key version merging, and delete processing into the C++ execution engine, StarRocks reduces Java↔C++ interactions, tightens the coupling between reading and subsequent vectorized computation, and reuses its I/O cache, memory tracker, and Query Profile infrastructure.

Three Reader Types and Selection Logic

StarRocks now has three readers for Paimon tables:

StarRocks File Reader — directly scans Parquet/ORC files when the task can be safely reduced to a file scan (common for append-only tables, but also possible for some primary-key tables).

Paimon C++ Reader — handles tasks that require primary-key version merging, delete-vector processing, and other Paimon-format semantics, bringing them into the native execution path.

JNI Reader — falls back to the Paimon Java SDK for system tables or other special cases; also available for compatibility checks and performance comparisons.

Reader choice is governed by the session variable paimon_reader_mode (introduced in v4.2): AUTO (default): prefers File Reader for scannable tasks; others stay on JNI. NATIVE: uses Paimon C++ Reader for supported tasks; system tables may still use JNI. JNI: forces Java SDK path for all applicable tasks.

Different scan tasks within the same SQL can use different readers. Note that AUTO does not automatically enable Paimon C++; explicit NATIVE is required for evaluation.

Enabling and Evaluating Native Reads

1. Create a Paimon Catalog (example uses Hive Metastore + HDFS via DLF):

CREATE EXTERNAL CATALOG paimon_lake
PROPERTIES (
  "type" = "paimon",
  "paimon.catalog.type" = "rest",
  "uri" = "https://cn-hangzhou-vpc.dlf.aliyuncs.com",
  "paimon.catalog.warehouse" = "<dlf_catalog_name>",
  "token.provider" = "dlf"
);

2. Verify the parameter exists and enable native mode with profiling:

SHOW VARIABLES LIKE 'paimon_reader_mode';
SET paimon_force_jni_reader = false;
SET paimon_reader_mode = 'NATIVE';
SET enable_profile = true;

paimon_force_jni_reader

is a higher-priority switch; it must be false for native reads to take effect.

3. Run existing SQL unchanged, e.g.:

SELECT order_date,
       SUM(amount) AS paid_amount
FROM paimon_lake.sales.orders
WHERE order_date = '2026-09-01'
  AND order_status = 'PAID'
GROUP BY order_date;

4. Compare NATIVE vs JNI modes under identical snapshots, SQL, and cluster resources. Keep cache conditions consistent (cold vs. hot runs) to isolate reader impact. Examine Query Profile and reader diagnostics to confirm the actual read path and inspect metrics such as scanned bytes, remote reads, cache hits, and memory usage. If bottlenecks lie in later join/aggregation stages, reader switching may yield limited end-to-end gains.

Performance Results

Tests on a single BE (8 cores, 32 GB RAM) showed Paimon C++ Reader delivering 1.4–3.4× speedups over JNI Reader:

Primary-key table count: 1,251 ms → 403 ms

1,000-column wide-table full aggregation: 62.0 s → 18.5 s

The advantage is most pronounced in primary-key merge and wide-table scenarios.

Future Roadmap

Expand native coverage — support more table types, data types, and schema-evolution cases while maintaining Paimon compatibility.

Further reduce overhead — optimize metadata preparation, remote I/O, merge logic, and columnar conversion; enhance cache utilization and observability.

Explore multi-modal and index acceleration — as Paimon adds BLOB, global indexes, vector, and full-text search, push filters and predicates down to lake indexes to cut data read volume.

Related code is merged into StarRocks main branch and will ship in the next release.