Why StarRocks Outperforms Traditional OLAP: Architecture, Storage Model, and Real‑World Use Cases

Why StarRocks

StarRocks is a new‑generation, ultra‑fast, full‑scenario MPP database designed to support a variety of analytical workloads with sub‑second query latency.

Simple architecture with a fully vectorized engine and a novel CBO optimizer for rapid query planning, especially multi‑table joins.

Strong real‑time analytics capabilities, supporting efficient queries on continuously updated data and modern materialized views.

Flexible modeling: users can build wide tables, star schemas, or snowflake schemas as needed.

MySQL‑compatible protocol, standard SQL support, no external dependencies, high availability, and easy operations.

System Architecture

The core processes are Frontend (FE) and Backend (BE). All nodes are stateful.

FE (Frontend) manages metadata, client connections, query planning, and scheduling.

Follower: participates in voting and query concurrency.

Leader: elected via a Paxos‑like BDBJE protocol; all transaction commits are initiated by the Leader.

Observer: does not participate in leader election; asynchronously syncs and replays logs to boost query concurrency.

BE (Backend) handles data storage and SQL execution.

Storage Architecture

In StarRocks, a table is split into multiple Tablets , each stored with multiple replicas on BE nodes.

Data distribution supports Hash and Range‑Hash (recommended). Range‑Hash first partitions then buckets the data, providing higher performance.

Range partitions can be added or removed dynamically.

Hash buckets are fixed after creation; only uncreated partitions can receive new bucket counts.

Choosing appropriate partition and bucket columns is critical for performance. Recommendations include using multi‑column bucketing for skewed data, aligning partitions with query predicates for high concurrency, and ensuring enough buckets to fully utilize CPU cores.

Key storage concepts:

Tablet : the smallest logical data unit; can be processed in parallel across machines.

Rowset : each data load creates a new version stored in a rowset.

Segment : large rowsets are split into segments for on‑disk storage.

Case Study 1 – Metric Factory Service

Background: The service collects business metrics in real time to support product status monitoring, anomaly detection, and alerting.

Requirements:

Full‑log detail, real‑time updates, hierarchical aggregation (day/week/month), high write throughput, configurable data retention, multi‑source data, and high‑concurrency queries.

Solution: Use Flink to consume Kafka streams and write to StarRocks in micro‑batches (10 s). StarRocks’ Flink connector allows flexible write rate control, balancing latency and load.

Data models used:

Detail model : stores raw, full‑detail events; supports high‑cardinality queries.

Aggregation model : pre‑aggregates metrics like PV/UV for fast reporting.

Update model : handles mutable data such as order status, with dynamic partitions for expiration.

Performance: StarRocks query for the whole period took 295 ms, while the previous MongoDB solution required 12 s (multiple queries + post‑processing).

Case Study 2 – Internal System Dashboard

Background: An internal dashboard provides project and task tracking for all employees, requiring frequent updates, multi‑table joins, and both hot and cold data access.

Original stack: MongoDB for task data and JSON model for flexibility, but complex queries took >10 s for large time‑range reports.

Migration to StarRocks:

Split the original collection into three tables using the detail model with daily partitions.

Store frequently updated dimension tables in MySQL and expose them as external tables in StarRocks.

Result: Single‑SQL aggregation queries now complete in sub‑second latency, simplifying development and improving performance.

Experience Sharing

Common issues and fixes encountered while using StarRocks:

Stream Load transaction limit : Exceeded default

max_running_txn_num_per_db</> (100). Increase the parameter or batch submissions.</li>
  <li><strong>FE file descriptor limit</strong>: "Too many open files" error. Raise the ulimit (e.g., <code>ulimit -n 65535

) in the FE startup script.

UDF Java errors : Missing JAVA_HOME when BE runs under supervisor. Add the environment variable.

Delete statement limitation : WHERE clause does not support BETWEEN. Use supported predicates (=, >, <, IN, etc.).

Routine Load group ID explosion : Specify a fixed group name when creating the routine load to avoid random group IDs.

Backend connection timeout : Routine Load failures can exhaust BrpcWorker threads. Pause problematic tasks to recover.

Future Plans

More business services will be migrated to StarRocks, replacing legacy OLAP engines and expanding use cases. Upcoming releases aim to reduce memory usage of primary‑key models, enhance column capabilities, improve bitmap query performance, and provide better multi‑tenant resource isolation. The team will continue contributing to the StarRocks community.