Skip to content
Xiaohongshu Cuts Vector Search Latency 61% with StarRocks-Lance Optimization
Databases 13 min read

Xiaohongshu Cuts Vector Search Latency 61% with StarRocks-Lance Optimization

Xiaohongshu optimized StarRocks querying of Lance-format training datasets, implementing TopN delayed materialization, scan range splitting, metadata caching, and balanced index scheduling, reducing cold vector search from 15s to 5.9s and hot from 4.5s to 1.5s on 7B rows.

StarRocks
StarRocks
StarRocks

Xiaohongshu's model teams need two query types on the same multimodal training datasets: SQL exploration (preview, filter, sort, aggregate) and vector similarity search (find candidate samples, then inspect content). As datasets grow to billions of rows with long text, dialogue arrays, and images, the OLAP engine team integrated StarRocks' SQL planning, distributed execution, and caching with the Lance data lake format to serve both workloads.

Performance Bottlenecks

Query profiles revealed two main bottlenecks: (1) read-side overhead from premature wide-column reads, repeated metadata loading/parsing, and poor cache reuse; (2) scheduling imbalance causing long tails when scan or index-retrieval tasks are unevenly distributed.

Optimization 1: TopN Delayed Materialization

For ORDER BY ... LIMIT queries returning a single page, cold reads took 7.5–11 seconds. The scan phase read far more data than the final result set because wide columns (long text, images) were decoded during candidate scanning even though they are only needed for display. StarRocks now splits reading into two phases: first read filter columns, sort columns, and RowId to compute global TopN; then re-read only the remaining columns for the selected RowIds. Both phases read the same dataset version, ensuring consistency between filtering/sorting and final sample content.

Figure 2: Two-phase read for TopN delayed materialization
Figure 2: Two-phase read for TopN delayed materialization

Optimization 2: Scan Range Splitting and Merging

Lance datasets have Fragments of varying sizes and counts. When data concentrates in a few large Fragments, scan work piles up on a few tasks, creating long tails. The Lance SDK added scan-range planning and range-read capability. StarRocks determines task granularity based on data volume, row count, and available parallelism. Large Fragments are split by row range across multiple parallel tasks; small Fragments are merged into a single task to reduce scheduling and initialization overhead.

Figure 3: Large Fragments split by row range, small Fragments merged
Figure 3: Large Fragments split by row range, small Fragments merged

Optimization 3: Read Path Optimization – Reducing Duplicate Initialization and Remote Reads

Cold reads require loading metadata for each scan and index-retrieval task. Concurrent tasks often re-read and re-parse the same metadata. For datasets with many Fragments, loading Fragment metadata prolongs task startup. Optimizations at three levels:

Manifest loading: Within a session, concurrent requests for the same dataset version's Manifest are merged; read and parse results are shared.

Fragment metadata (RED layout): Support grouped, paged, range-based reads with controlled concurrency, reducing wait and repeated file-header reads.

Index initialization: Merge concurrent load requests; read only the relevant index's auxiliary metadata and reuse file metadata already parsed when opening the index.

On the StarRocks side, scan tasks reuse a fixed-version Dataset object and its metadata. For Java Reader scans, enabling scan output cache allows Data Cache to reuse cached data batches when dataset version, scan parameters, and output columns match, avoiding repeated reads and decoding.

Figure 4: Manifest, Fragment, and index metadata loading optimization
Figure 4: Manifest, Fragment, and index metadata loading optimization

Optimization 4: Vector Retrieval – Balancing Load and Cache Reuse

Index cache hit rate directly affects vector search latency. Reusing loaded indexes reduces remote reads and initialization. However, the original hash scheduling concentrated index segments on the same node for cache affinity, causing hotspots when many segments landed on one node. StarRocks now sets a per-node index-segment quota. Within the quota, the original hash assignment is kept; excess segments are assigned to nodes with remaining capacity, further balanced by the row count each segment covers.

During execution, each BE retrieves its assigned index segments in parallel. The execution layer merges candidate results into a global TopK. Each BE reuses the Lance SDK session and hot index cache, merges concurrent loads for the same cache item, and uses Data Cache to reduce repeated OSS access for index files.

Figure 5: Index segment scheduling balancing load and cache reuse
Figure 5: Index segment scheduling balancing load and cache reuse

Business Results

Dataset Exploration: 4 Statistical Queries 82–90% Faster

Four high-frequency, long-running statistical queries covering mixed dialogue, long-text QA, instruction data, and math dialogue were tested. After optimization, all four cold-read latencies dropped below 2.4 seconds, with three under 1 second.

Figure 6: Cold-read latency comparison for four representative statistical queries
Figure 6: Cold-read latency comparison for four representative statistical queries

Vector Retrieval: Cold Query 15s → 5.9s, Hot Query 4.5s → 1.494s

Tested on an internal 7-billion-row, 2 TB training dataset with 1024-dim vectors, approximate cosine distance, Top-100 recall, concurrency 10. Cold-query average latency fell from 15 seconds to 5.9 seconds (~61% reduction). Hot-query average latency fell from 4.5 seconds to 1.494 seconds (~67% reduction).

Figure 7: Vector retrieval cold and hot average latency before and after optimization
Figure 7: Vector retrieval cold and hot average latency before and after optimization

Future Evolution: Toward 10-Billion-Scale Vector Search

Xiaohongshu now runs StarRocks and Lance in production, providing low-latency dataset exploration and multimodal retrieval (preview, filter, statistics, vector search). For 10-billion-scale vector search, the team plans to validate distributed execution, cache efficiency, and concurrency on larger indexes, evaluating recall, latency, and resource cost to support model teams discovering and filtering training samples in even larger datasets. They also intend to contribute production issues, optimizations, and validation experience back to the StarRocks and Lance communities.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

StarRocksOLAPmetadata cachingvector retrievaldata lakeLancescan schedulingTopN optimization
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.