StarRocks Embraces Paimon 2.0: Unifying Columnar Analytics and Vector Search in a Single SQL Engine

StarRocks has extended its lakehouse support to Paimon 2.0, targeting workloads that blend traditional structured analytics with multimodal retrieval — video frames, sensor streams, embeddings — while keeping data on object storage. The article walks through the motivation, the Paimon 2.0 primitives, the resulting query-execution changes in StarRocks, operational best practices, and the roadmap for mixed-workload isolation.

Why Multimodal Lakehouse?

Classic lakehouses manage structured tables (orders, logs, tags). New AI-driven scenarios — autonomous driving, embodied robotics, ad creative analysis, game asset governance — require joining raw media (video, point clouds, images) with business metadata and model-derived features (labels, scores, vectors). Public datasets illustrate the scale: Waymo Open Motion Dataset (103,354 20-second scenes linking trajectories, maps, image features) and Google DeepMind's Open X-Embodiment (1M+ episodes across 22 robot types). In each case, a single observation is meaningless without its temporal context, environment, and task goal.

Paimon 2.0 Core Primitives

The article focuses on append-only tables with Row Tracking and Data Evolution enabled .

Row Tracking assigns a stable row_id at first write. Raw media (BLOB) and business attributes (timestamp, vehicle model) land in columnar data files; the row_id becomes the primary key for later linkage.

Data Evolution allows subsequent pipelines to backfill derived columns (labels, evaluation scores, vectors) by writing new column files that carry the same row_id. Unchanged columns and BLOBs are reused — no full-row rewrite. Vectors are stored in dedicated VECTOR files.

Global Index (scalar BTree/Bitmap, vector ANN, full-text) is built asynchronously on chosen columns. Only indexes valid for the query's snapshot participate. At read time, Paimon merges column files and indexes by row_id to reconstruct the logical row.

A concrete driving-frame example shows the lifecycle: initial write stores timestamp + vehicle_type + BLOB reference; later a perception model writes label=lane_departure, score=0.92, and a 256-dim embedding vector — all tied to the original row_id.

From OLAP Scan to Search: Two Access Paths in One Engine

StarRocks already handled metadata pruning → parallel columnar scan → filter/join/aggregate. Paimon 2.0 adds an index-driven path:

OLAP path: partition pruning → file-level min/max skipping → row-group/page statistics → parallel column decode → relational operators. Optimized for large-range scans, reporting, broad joins.

Search path: index lookup (scalar → row_id set; vector/full-text → row_id + distance/score) → candidate merge/ranking (AND/OR logic, score fusion) → row_id-based fetch of projected columns → relational operators. Optimized for needle-in-haystack retrieval.

The engine chooses per query. Cost model now accounts for index search scale, candidate count, hit-file distribution, and Data Evolution column-merge overhead. Index access is not always cheaper: many candidates scattered across thousands of files can exceed a sequential scan cost.

Execution Adaptations for Search

Co-scheduling: Vector/full-text indexes are sharded; search tasks and subsequent fetch tasks have different parallelism boundaries. StarRocks schedules them separately, merging fetch requests by target file to reuse metadata/data cache.

Candidate merge + late materialization: Small row_id + score payloads shuffle globally for top-K ranking. Heavy columns (media URLs, long text) are fetched only for final top-K rows. However, columns needed for pre-ranking filters or joins must be materialized earlier; raw vectors for exact re-ranking may also need early fetch.

Unified cost-based path selection: CBO evaluates end-to-end cost of scan vs. index paths, including ANN recall/latency trade-offs.

Operational Best Practices (from production at Alibaba/Alibaba Cloud)

Write & Compaction: Partition by frequent filters (time, domain). Target ~256 MiB files (1 TiB @ 1 MiB → 1M objects; @ 256 MiB → ~4K objects). Monitor file count/size distribution, compaction backlog, and correlate with StarRocks scan bytes, object requests, cache hit rate. Row Tracking append tables differ from primary-key tables — bucket strategies don't transfer directly.

Data Evolution: Backfill only changed feature columns; record model version, compute time, status. Different model versions produce incompatible vector spaces — business side must govern comparability. Coordinate backfill, compaction, and index refresh to avoid read-amplification from many small column files.

Global Index Config: Choose index type by query selectivity, data scale, coverage needs, recall/latency SLA — not just column type. Tune index_shard count: too many → merge overhead; too few → hot shards. Shard strategy differs per index type (BTree vs. ANN) and is independent of table partitioning/bucketing. Validate end-to-end latency (cold/hot cache) at fixed model version, distance metric, recall target.

DLF Managed Maintenance: Alibaba Cloud DLF handles async index build, compaction, snapshot cleanup, partition TTL, orphan file GC. Treat "data written" and "index queryable" as separate observability milestones. Tune maintenance frequency against query latency, scan volume, cache hit rate, and storage bandwidth contention.

Roadmap: Mixed-Workload Optimization

Compute-group isolation: Separate compute groups for low-latency retrieval (stable tail latency) vs. high-throughput batch analytics, each with independent concurrency limits, resource quotas, autoscaling. Scheduler must also account for shared object-storage bandwidth between foreground queries and background compaction/index-build jobs.

Index-aware planning: Query snapshot may have partial index coverage. Planner must either schedule supplemental scans for unindexed portions or expose the searchable range to the application, letting users trade completeness for freshness/latency.

Query metrics driving maintenance: Feed StarRocks latency, read volume, cache hit rate into compaction/index-build prioritization. For operations that reassign row_id or mutate indexed columns, coordinate index rebuild and query cutover to preserve correctness.

The piece closes by positioning StarRocks × Paimon as a foundational AI Data Infra layer where data management, content retrieval, and relational analysis converge on a single lake copy.