Xiaohongshu Cuts Vector Search Latency 61% with StarRocks-Lance Optimization
Xiaohongshu optimized StarRocks querying of Lance-format training datasets, implementing TopN delayed materialization, scan range splitting, metadata caching, and balanced index scheduling, reducing cold vector search from 15s to 5.9s and hot from 4.5s to 1.5s on 7B rows.
Xiaohongshu's model teams need two query types on the same multimodal training datasets: SQL exploration (preview, filter, sort, aggregate) and vector similarity search (find candidate samples, then inspect content). As datasets grow to billions of rows with long text, dialogue arrays, and images, the OLAP engine team integrated StarRocks' SQL planning, distributed execution, and caching with the Lance data lake format to serve both workloads.
Performance Bottlenecks
Query profiles revealed two main bottlenecks: (1) read-side overhead from premature wide-column reads, repeated metadata loading/parsing, and poor cache reuse; (2) scheduling imbalance causing long tails when scan or index-retrieval tasks are unevenly distributed.
Optimization 1: TopN Delayed Materialization
For ORDER BY ... LIMIT queries returning a single page, cold reads took 7.5–11 seconds. The scan phase read far more data than the final result set because wide columns (long text, images) were decoded during candidate scanning even though they are only needed for display. StarRocks now splits reading into two phases: first read filter columns, sort columns, and RowId to compute global TopN; then re-read only the remaining columns for the selected RowIds. Both phases read the same dataset version, ensuring consistency between filtering/sorting and final sample content.
Optimization 2: Scan Range Splitting and Merging
Lance datasets have Fragments of varying sizes and counts. When data concentrates in a few large Fragments, scan work piles up on a few tasks, creating long tails. The Lance SDK added scan-range planning and range-read capability. StarRocks determines task granularity based on data volume, row count, and available parallelism. Large Fragments are split by row range across multiple parallel tasks; small Fragments are merged into a single task to reduce scheduling and initialization overhead.
Optimization 3: Read Path Optimization – Reducing Duplicate Initialization and Remote Reads
Cold reads require loading metadata for each scan and index-retrieval task. Concurrent tasks often re-read and re-parse the same metadata. For datasets with many Fragments, loading Fragment metadata prolongs task startup. Optimizations at three levels:
Manifest loading: Within a session, concurrent requests for the same dataset version's Manifest are merged; read and parse results are shared.
Fragment metadata (RED layout): Support grouped, paged, range-based reads with controlled concurrency, reducing wait and repeated file-header reads.
Index initialization: Merge concurrent load requests; read only the relevant index's auxiliary metadata and reuse file metadata already parsed when opening the index.
On the StarRocks side, scan tasks reuse a fixed-version Dataset object and its metadata. For Java Reader scans, enabling scan output cache allows Data Cache to reuse cached data batches when dataset version, scan parameters, and output columns match, avoiding repeated reads and decoding.
Optimization 4: Vector Retrieval – Balancing Load and Cache Reuse
Index cache hit rate directly affects vector search latency. Reusing loaded indexes reduces remote reads and initialization. However, the original hash scheduling concentrated index segments on the same node for cache affinity, causing hotspots when many segments landed on one node. StarRocks now sets a per-node index-segment quota. Within the quota, the original hash assignment is kept; excess segments are assigned to nodes with remaining capacity, further balanced by the row count each segment covers.
During execution, each BE retrieves its assigned index segments in parallel. The execution layer merges candidate results into a global TopK. Each BE reuses the Lance SDK session and hot index cache, merges concurrent loads for the same cache item, and uses Data Cache to reduce repeated OSS access for index files.
Business Results
Dataset Exploration: 4 Statistical Queries 82–90% Faster
Four high-frequency, long-running statistical queries covering mixed dialogue, long-text QA, instruction data, and math dialogue were tested. After optimization, all four cold-read latencies dropped below 2.4 seconds, with three under 1 second.
Vector Retrieval: Cold Query 15s → 5.9s, Hot Query 4.5s → 1.494s
Tested on an internal 7-billion-row, 2 TB training dataset with 1024-dim vectors, approximate cosine distance, Top-100 recall, concurrency 10. Cold-query average latency fell from 15 seconds to 5.9 seconds (~61% reduction). Hot-query average latency fell from 4.5 seconds to 1.494 seconds (~67% reduction).
Future Evolution: Toward 10-Billion-Scale Vector Search
Xiaohongshu now runs StarRocks and Lance in production, providing low-latency dataset exploration and multimodal retrieval (preview, filter, statistics, vector search). For 10-billion-scale vector search, the team plans to validate distributed execution, cache efficiency, and concurrency on larger indexes, evaluating recall, latency, and resource cost to support model teams discovering and filtering training samples in even larger datasets. They also intend to contribute production issues, optimizations, and validation experience back to the StarRocks and Lance communities.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
