StarRocks: Integrated vs Separated Storage-Compute Read/Write Flows

Part 1: Integrated Storage-Compute Architecture (Classic)

In the integrated architecture, compute and storage are tightly coupled in BE (Backend) nodes. Data is stored directly on BE local disks (or locally mounted distributed file systems), while metadata is managed centrally by the FE (Frontend) .

Write Principle (Stream Load Example)

Core flow: Data is written to BE local disk using "local I/O + batch merge" for efficiency.

FE Scheduling: Client sends write request (e.g., Stream Load). FE parses request, determines target BE nodes based on table bucketing rule ( DISTRIBUTED BY HASH) and load balancing, and syncs table metadata (partitions, encoding).

BE Receive and Buffer: Target BE receives batch data, writes to in-memory buffer ( MemTable). When data reaches threshold (e.g., 128 MB) or timeout (e.g., 5 minutes), a "flush" operation encodes data into columnar format (StarRocks' custom Columnar format) and writes to a local temporary directory.

File Merge (Compaction): BE background process periodically merges small files ( Compaction), e.g., merging ten 10 MB files into one 100 MB file, reducing subsequent query I/O.

Metadata Update: After file write completes, BE reports file path and size to FE. FE updates metadata (file index, partition version) and commits the write transaction.

Key Characteristics:

Data written directly to local disk, avoiding remote network I/O; write latency low (milliseconds).

Relies on BE-local Compaction to merge small files, optimizing query performance.

Storage and compute are bound; scaling BE nodes requires adding both compute and storage resources.

Read Principle (SQL Query Example)

Core flow: Compute and data are co-located, minimizing cross-node data transfer.

FE Generates Execution Plan: Client sends SQL. FE parses and optimizes (predicate pushdown, partition pruning), determines which BE nodes hold the data (data locality principle, preferring BEs that store the target data).

BE Local Compute: Target BE receives plan, reads corresponding data files from local disk (leveraging OS cache), and performs filtering ( WHERE), aggregation ( GROUP BY), joins, etc., locally.

Result Aggregation and Return: If multiple BEs are involved (e.g., cross-bucket aggregation), FE designates one BE as coordinator to aggregate intermediate results, compute final result, and return to client.

Key Characteristics:

Compute and data on same node, avoiding remote data pulls; query latency low (especially for small tables).

Relies on BE local disk I/O and cache; large-table query performance limited by single-node disk bandwidth.

Limited scalability (requires scaling entire BE cluster, high cost).

Part 2: Separated Storage-Compute Architecture

In the separated architecture, compute and storage are fully decoupled: compute is handled by CN (Compute Node) , storage relies on remote object storage (e.g., S3, OSS), and metadata is still managed by FE (with added object storage file indexes).

Write Principle (Stream Load Example)

Core flow: Data is processed by CNs and then uploaded to object storage in batches, adapting to object storage characteristics.

FE Scheduling: Client sends write request. FE selects a CN as coordinator, syncs table metadata (bucket rules, object storage paths).

CN Data Processing: Coordinator CN receives batch, splits data by bucket rules, distributes shards to target CNs. Target CNs encode data (e.g., ZSTD compression), convert to columnar format, and temporarily cache in local memory or disk (not persisted).

Batch Upload to Object Storage: When a CN's cached data reaches threshold (e.g., 100 MB), it merges into a large file and uploads via S3/OSS SDK to object storage (path specified by FE metadata, e.g., s3://starrocks-data/db1/tbl1/partition=202408/).

Metadata Update: After all CNs finish upload, coordinator reports to FE. FE updates metadata (adds object storage file index, version) and commits transaction.

Key Characteristics:

Data ultimately resides in object storage; CNs only cache temporarily, no local persistence.

Relies on "batch upload of large files" to reduce object storage metadata overhead (object storage is inefficient with small files).

Compute (CN) and storage (object storage) can scale independently (CN for compute, object storage for capacity).

Read Principle (SQL Query Example)

Core flow: CN pulls data from object storage into local cache, computes, and returns results.

FE Generates Execution Plan: Client sends SQL. FE parses and optimizes, uses metadata (object storage file indexes, bucket info) to generate plan, assigns to CNs based on load balancing.

CN Pulls Data: Target CN receives plan, pulls data from object storage via SDK using FE-provided paths (e.g., s3://.../file1) into local memory cache (or temporary disk), preferring cache to avoid repeated pulls.

CN Compute and Result Aggregation: CN executes filtering, aggregation locally on pulled data. If multiple CNs involved, FE designates one CN to aggregate intermediate results and return final result to client.

Key Characteristics:

Data must be pulled from object storage to CN (cross-network I/O); query latency slightly higher than integrated, especially on first query without cache .

Relies on CN local cache (memory, SSD) to accelerate repeated queries; large-table query performance limited by network bandwidth and object storage throughput .

High elasticity (CNs can scale elastically for compute peaks, object storage scales on demand).

Part 3: Core Differences Comparison

Summary:

Integrated architecture achieves low-latency reads/writes through compute-storage localization, suitable for small-to-medium data and low-latency scenarios, but scalability is limited.

Separated architecture achieves elastic scaling through compute-storage decoupling, suitable for large-scale data scenarios, but depends on network and object storage performance, requiring batch read/write and cache optimizations.

The choice between architectures should consider data volume, latency requirements, and scaling needs: prefer integrated for small-scale low-latency; prefer separated for large-scale elastic scenarios.