How StarRocks’ Compute‑Storage Separation Cuts Costs and Boosts Query Efficiency

Background

Enterprises need to reduce cost and improve efficiency in analytics. Traditional monolithic compute‑storage architectures waste resources because cold data is stored together with hot data, leading to high hardware costs and limited elasticity.

Key Challenges

Real‑time ingestion and long‑term retention: log volumes can reach hundreds of terabytes to petabytes and must be kept for months or years.

Hot vs. cold access patterns: recent data (minutes to days) is queried frequently with low latency, while older data is accessed rarely.

Cost‑Saving Example – Log Analysis

Customer A generates 1 TB of incremental logs per day and retains them for two years. In a traditional integrated architecture each server provides 20 TB storage, three‑way replication and 50 % compression result in ~700 TB of stored data, requiring about 52 servers. Because 80 % of queries target the most recent 7 days, the hot workload actually needs only ~0.52 servers. By scaling the hot‑data workload elastically, the cost ratio becomes (0.2 × 52 + 0.8 × 1) / 52 ≈ 0.22, i.e., a 78 % hardware‑cost reduction (≈11 servers instead of 52).

Efficiency Gains – E‑commerce Reporting

In a SaaS e‑commerce scenario daily query QPS peaks around 100. A three‑node cluster (32 vcores, 128 GB each) handles normal load, but month‑end aggregation pushes load to 100 % and P99 latency > 10 s. With compute‑storage separation the cluster can be expanded to 14 nodes only during the peak week, giving an average effective node count of 5.75 and a 2.43× overall query‑performance improvement.

StarRocks 3.0 Compute‑Storage Separation

StarRocks 3.0 introduces a unified storage‑access layer (StarOS) that supports S3‑compatible object stores (OSS, MinIO) and HDFS. Data files keep the same segment format, so existing indexes are reused.

Storage Layer

Compute nodes are stateless; all data resides in external object storage or HDFS, dramatically lowering storage cost and enabling independent scaling of compute resources.

Local‑Disk Cache for Hot Data

Hot data can be cached on each compute node’s local disks. The following table‑creation parameters control the cache behavior:

enable_storage_cache   // true to enable local‑disk caching for hot data
storage_cache_ttl      // lifetime of cached hot data (seconds)
enable_async_write_back // true: write returns after local write, background task pushes data to object storage

When enable_async_write_back is true, writes are acknowledged after the local disk persists the data; a background task asynchronously uploads the data to the object store, improving write throughput at the cost of a small reliability risk.

During query execution, hot data is served from the local cache, while cold data is fetched from object storage and optionally prefetched to reduce latency.

Multi‑Version Data Management

Each data load creates an immutable version. TabletMeta and TabletData files are stored in the backend; TabletMeta records indexes for all segment files of that version. This immutable versioning lays the groundwork for future features such as Time‑Travel queries.

Compaction Scheduling

Compaction tasks are now scheduled by the Frontend (FE) node, which can dispatch them to any Backend (BE) node or to a dedicated compaction cluster. This separation prevents compaction from interfering with user queries and allows dynamic scaling of the compaction resource pool.

Performance Evaluation

Data Import (ClickBench)

Test configuration: 1 FE + 3 BE, 160 GB CSV (1 GB per file). Throughput was measured under different client concurrency levels.

Client concurrency | Integrated time (s) / Cloud‑native time (s) | Integrated throughput (MB/s) / Cloud‑native throughput (MB/s)
1                  | 1007 / 1024                                 | 162 / 160
4                  | 377  / 379                                  | 434 / 432
16                 | 363  / 183                                  | 451 / 895

As concurrency increases, the cloud‑native version’s write throughput continues to rise until the network bandwidth becomes the bottleneck, outperforming the integrated version.

Query Performance (TPC‑DS 1 TB)

Three deployment modes were compared:

sr‑3.0‑native (integrated)

sr‑3.0‑cloud‑native (full cache hit)

sr‑3.0‑cloud‑native‑no‑cache (all caches disabled)

Results:

Mode                     | Total query time (s)
sr‑3.0‑native            | 423
sr‑3.0‑cloud‑native      | 428  (full cache hit, comparable to integrated)
sr‑3.0‑cloud‑native‑no‑cache | 668  (cold‑data access, still acceptable)

Even without any cache, StarRocks uses prefetching and a multi‑layered storage hierarchy to keep performance degradation within a reasonable range.

Conclusion

StarRocks 3.0’s compute‑storage separation can achieve up to 78 % hardware‑cost reduction and more than 2× query‑performance improvement for typical hot‑cold workloads, while preserving the high‑speed analytical capabilities of the original MPP engine. Ongoing work includes multi‑cluster isolation, write‑ahead‑log (WAL) storage, and further elastic resource management.