How StarRocks Achieves Lightning‑Fast Data Lake Analytics

Overview

StarRocks is a high‑performance analytical database that combines a minimal architecture, a vectorized execution engine, and a cost‑based optimizer (CBO) to achieve fast query processing, especially for multi‑table joins.

System Architecture

FE (Frontend) : Parses SQL, performs semantic analysis, generates logical and physical plans, and creates execution fragments.

BE (Backend) : Executes fragments in parallel, reads data from storage, applies filters, aggregates, and returns results.

Data‑lake components include Table Format (metadata), File Format (e.g., Parquet, ORC, Avro), and Storage (HDFS, OSS, S3).

Technical Details

CBO Optimizer

The optimizer is built on the Cascades and ORCA frameworks and supports TPC‑DS queries, expression and CTE reuse, join reordering, runtime filter push‑down, and low‑cardinality dictionary optimization. Accurate cost estimation depends on up‑to‑date table‑ and column‑level statistics, which can be collected automatically or manually in full or sample mode.

MPP Execution

StarRocks splits a query into multiple Query Fragments . Each fragment contains operators such as Scan, Filter, and Aggregate and is scheduled to BE nodes. Fragments run in parallel pipelines, avoiding stage‑by‑stage execution. Shuffle operations enable horizontal scaling for large joins and high‑cardinality aggregations.

Vectorized Execution Engine

All operators are reimplemented to process column‑wise batches. Vectorized execution reduces virtual‑function calls and branch mispredictions, improves CPU cache utilization, and enables SIMD optimizations, delivering a 5‑10× performance boost over row‑wise execution.

Data Lake Query Support

Iceberg v1 table queries – https://github.com/StarRocks/starrocks/issues/1030

Hive external table queries – https://docs.starrocks.com/zh-cn/main/using_starrocks/External_table

Hudi COW table queries – https://github.com/StarRocks/starrocks/issues/2772

Optimization techniques applied to lake queries include:

Statistics collection (full or sample) for accurate cost estimates.

Partition pruning to scan only relevant partitions.

Join reorder based on data size and distribution.

Predicate push‑down (e.g., Min/Max filters on Parquet) to filter data at the storage layer.

MySQL [hive_test]> explain select l_quantity from lineitem;

create external table store_sales(
  ss_sold_time_sk bigint,
  ss_item_sk bigint,
  ...
) ENGINE=HIVE PROPERTIES (
  "resource" = "hive_tpcds",
  "database" = "tpcds",
  "table" = "store_sales"
);

select ss_sold_time_sk from store_sales
where ss_sold_date_sk between 2451911 and 2451941
order ss_sold_time_sk;

Benchmark Results

Using the TPC‑H 100 GB dataset (22 queries), StarRocks was compared with Trino. Three execution modes were measured:

StarRocks local storage: total time 21 s

StarRocks on Hive (external table): total time 92 s

Trino on Hive: total time 307 s

The additional latency for Hive queries is mainly due to network I/O and remote storage access. Future work includes caching mechanisms to narrow the gap.

Future Roadmap

Integrate a push‑based pipeline execution engine to further reduce query latency.

Automate hot‑cold data tiering, moving infrequently accessed data from local tables to the lake.

Eliminate explicit external‑table creation by syncing lake resources automatically.

Support advanced lake features: Hudi MOR tables, Iceberg v2 tables, direct writes to the lake, time‑travel queries, and richer catalog integration.

Introduce hierarchical caching to boost lake‑query performance.