Skip to content
StarRocks × Lance: Distributed Vector Search & Global Late Materialization for Multimodal Lakehouse
Databases 21 min read

StarRocks × Lance: Distributed Vector Search & Global Late Materialization for Multimodal Lakehouse

The article details StarRocks' integration with the Lance columnar format for multimodal lakehouse workloads, covering distributed index search, two materialization strategies (non-GLM and Global Late Materialization), and a 2-billion-vector case study achieving 40x latency reduction and 16x throughput over Ray+Lance.

StarRocks
StarRocks
StarRocks

Multimodal lakehouse architectures store data in open formats on object storage, allowing feature engineering, model training, retrieval, and analytical engines to share the same data. StarRocks integrates with Lance — a columnar format designed for AI workloads that combines columnar scans, random access, data evolution, and hybrid search — to unify vector retrieval with SQL-based filtering, joins, and aggregation in a single query plan.

Lance Data Organization and Access

Lance separates index files from column data files, enabling index upgrades or quantization changes without rewriting base data. Queries first locate candidate records via the index, then fetch required columns on demand. The page organization and encoding of column files directly affect the amount of data read and decoded during this second phase.

StarRocks Integration Architecture

The integration splits work into index search and column reading, sharing a fixed query version but allowing different parallelism. The Frontend (FE) pins the data version and index coverage, generates search tasks, and after search produces candidate records, groups read tasks by fragment, file, and required columns. For full-coverage queries, indexed portions use index search while uncovered fragments fall back to supplemental scans. The scheduler assigns tasks to Backend (BE) nodes based on load and cache distribution.

Execution-Node Division of Labor

Lance SDK (Rust) handles per-task filtering, index search, and column reads using DataFusion for columnar/streaming execution. The C++ BE manages the pipeline, exchanges, and downstream relational operators. An adaptation layer converts SDK columnar batches into StarRocks Chunks for joins, aggregations, and sorts.

Key Integration Challenges

Rust–C++ columnar execution bridging: SDK plans filtering, search, and reads; DataFusion executes; C++ BE runs pipeline and exchanges. Adaptation layer converts SDK batches to Chunks.

Unified memory management: Rust temporary buffers, Arrow batches, in-flight reads, and Chunk conversions share a single query memory budget. Cross-language data ownership and release timing are explicit. Arrow buffers are reused when layouts match; otherwise conversion cost is accounted. Cross-query index and metadata caches are managed per-BE.

Thread, coroutine, and I/O co-scheduling: Index search parallelism, Rust async task count, and object-storage I/O concurrency are co-designed. Batch buffering and backpressure match BE pipeline consumption. Budgets for in-flight requests and prefetch depth prevent thread oversubscription, buffer bloat, and async-read memory pressure.

Cache affinity and load correction: Each BE maintains independent index and metadata caches. Immutable objects are reused by identity; row-location and visibility data bind to the query snapshot. The scheduler picks execution nodes using cache distribution, task size, and node load, preferring cache reuse and spreading hotspots.

Two Materialization Strategies for Distributed Top-K

In distributed Top-K retrieval, each BE produces local candidates, then a global merge determines the final Top-K. Wide business columns (trajectory details, long text) can be read at the source BE or after global Top-K is fixed.

Non-GLM (materialize at source): Source BEs fetch needed columns, form Chunks, and send them via cross-node Exchange. If the reader supports staged column reads, local filtering or Top-K can narrow candidates before reading output columns. This reuses source-side reads and caches, avoiding extra round-trips. Preferred when output columns are narrow, global merge discards few candidates, or row-location/RPC cost is high.

Global Late Materialization (GLM): BEs first exchange only row IDs, scores, and essential columns. After global filtering/Top-K narrows candidates, row positions are resolved under a fixed snapshot, grouped by fragment and file, and required columns are batch-fetched. Fetched columns are aligned with hit records, assembled into Chunks, and fed to downstream operators. This reduces useless I/O, decoding, and network transfer when delayed columns are wide and many candidates are discarded. Row-location, RPC, and fetch scheduling costs are accounted.

Both paths respect reader capabilities and query semantics: columns used for candidate qualification, ranking, or refine must be read before the respective operation; columns used only for projection or later analysis can be deferred. Both integrate with joins, expression evaluation, and aggregation, with the execution plan governing read placement and subsequent computation.

Case Study: 2-Billion-Vector Hybrid Search and Analysis

A large e-commerce trajectory platform manages 2 billion 1024-dimensional Float32 vectors across 500 fragments, organized into 100 index segments. Queries combine attribute filtering with vector similarity, then join business records for analysis, returning wide trajectory columns.

Index Selection and Refine

Index choice balances capacity and recall; refine usage depends on per-query latency and accuracy targets. StarRocks supports query-level refine settings on the same index.

High QPS, low latency: Real-time search uses IVF_SQ with TopK=1000 and refine disabled, avoiding raw-vector reads and distance recomputation.

Higher recall and ranking quality: Refine can be enabled (typically starting at refine factor=4) and tuned per observed recall and latency. Refine only re-ranks already-found candidates; recall remains bounded by index search scope.

Storage breakdown (including 8-byte row IDs): SQ uses 8 bits/dimension; PQ uses 128 8-bit codes/vector. SQ vectors + row ID payload = 2.064 TB; adding raw vectors brings total to 10.256 TB. IVF centroids, quantization parameters, and file metadata are additional.

Wide-Column Return: Choosing GLM

Assuming 8 KiB per row for wide business columns, the article compares payloads for two materialization paths. 100 independent search tasks each return local Top-1000, producing 100,000 candidates, from which global Top-1000 is selected. Columns used for filtering and ranking are excluded from delayed payload.

Non-GLM: materialize all 100,000 candidates → 819.2 MB payload.

GLM: materialize only 1,000 global winners → 8.192 MB payload, a 99% reduction in deferrable-column payload.

The case adopts GLM: exchange row-location info and scores, determine global Top-K, then fetch wide columns for winners, assemble Chunks, and proceed to downstream analysis. Actual I/O savings depend on encoding, caching, and batch request efficiency; the 99% figure refers to payload reduction, not end-to-end speedup.

Business Impact: Single SQL from Offline Batch to Real-Time Ad-Hoc

The original pipeline used Ray + Lance for Top-K extraction, then handed off to StarRocks for joins and aggregation, requiring batch handoffs. After integration, StarRocks executes hybrid search, Top-K, on-demand column reads, and OLAP analysis in one SQL statement, enabling direct ad-hoc analysis and eliminating intermediate data movement and wait times. According to Alibaba Cloud tests on the customer workload, StarRocks reduced end-to-end data-ready-for-analysis latency to 1/40 of the original (minutes to seconds) and achieved 16× the hybrid-search throughput of Ray + Lance.

Multimodal Lakehouse as Systems Engineering: Ongoing Optimizations

Sustaining performance, cost, and data freshness requires coordinating ingestion, feature backfill, index building, and query serving.

Joint management of fragments, index segments, and incremental coverage: Avoid small-file and incremental-segment accumulation while balancing search parallelism, maintenance task size, and load distribution. After compaction or index rebuild, update coverage ranges and row-location info per new version, preserve reusable object caches, and manage data visibility and index searchability separately.

Separate observation of search, refine, exchange, and delayed column reads: Query metrics break down candidate counts, search latency, raw-vector read volume, exchange bytes, and fetch/lookup latency. Cache hits for index and data reads are tracked separately, then correlated with full query latency to evaluate optimization gains.

Reliable cost inputs for the optimizer: Accumulate filter selectivity, local/global shrink ratios, column widths, and batch-read costs to refine materialization-strategy cost models. Comparisons between non-GLM and GLM must fix model, distance metric, recall target, candidate budget, and cover varying K, column widths, cache temperatures, and concurrency levels for apples-to-apples results.

Compute groups and background scheduling for mixed workloads: Interactive retrieval, batch analysis, index maintenance, and feature engineering have different resource and latency needs. StarRocks compute groups isolate retrieval and analysis workloads with separate concurrency and resource budgets. Background tasks coordinate shared object-storage bandwidth and request pressure to control tail-latency variance.

StarRocks plans to contribute the core Lance integration code to the open-source community, inviting developers and users to test, provide feedback, and contribute code to advance multimodal analytics and retrieval capabilities.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

StarRocksvector searchOLAPdistributed SQLcolumnar formatLancemultimodal lakehouselate materialization
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.