HELMSMAN Redefines Large-Scale Vector Retrieval with High-Performance ANNS

Approximate nearest neighbor search (ANNS) is a core capability for search, recommendation, advertising, and content safety systems, but the traditional in‑DRAM graph indexes (e.g., HNSW) face unsustainable cost growth as vector data scales to hundreds of billions of vectors and petabyte‑level DRAM usage.

Modern NVMe SSDs offer high bandwidth at a fraction of DRAM cost, yet simply moving data to SSDs does not work because graph‑based search relies on serial I/O dependencies that cannot fully utilize SSD bandwidth and amplify tail latency.

The Xiaohongshu engine architecture team proposes HELMSMAN, a full‑flash, high‑performance ANNS system that replaces graph indexes with a clustering‑based approach. Queries first locate nearest centroids in memory, then batch‑read the corresponding cluster lists from SSD, enabling independent I/O operations that fully exploit SSD bandwidth.

HELMSMAN addresses three major shortcomings of prior SSD‑DRAM hybrid systems (DiskANN, SPANN, etc.): insufficient throughput due to Linux I/O overhead, non‑adaptive search strategies that either over‑scan or under‑scan clusters, and slow index construction that cannot keep up with frequent model updates.

Key innovations include:

Custom ANNS storage stack: Built on SPDK, the user‑space stack bypasses the kernel file system and NVMe driver, managing NVMe queues directly and storing fixed‑size cluster lists in raw logical blocks for single‑shot I/O.

Leveling‑Learned Search Pruning (LLSP): A router model predicts the appropriate nprobe based on query difficulty and top‑k, followed by a GBDT model that decides the exact number of clusters to read, ensuring adaptive pruning before any SSD access.

GPU‑accelerated distributed construction pipeline: Three stages—GPU‑accelerated coarse k‑means, elastic CPU‑based cluster partitioning and padding, and multi‑core CPU merging with LLSP training—run on a RED‑Ray and Daft execution backbone, achieving hour‑level rebuilds for 0.1 B‑scale indexes and a few hours for 10 B‑scale indexes.

Experimental results on public benchmarks and Xiaohongshu production workloads show 2–16× throughput improvements over DiskANN, Starling, PipeANN, and SPANN, reaching up to 85% of pure‑memory HNSW throughput while maintaining 5–10 ms average latency and strict tail‑latency guarantees. SSD bandwidth utilization rises to ~85% on PCIe Gen4 and ~70% on Gen5. Construction time for a 0.1 B index drops below one hour, and a 10 B index rebuild completes in 4–7 hours, a ~10× speedup.

In production, HELMSMAN runs on ~40 all‑flash servers, replacing a legacy deployment that required ~35,000 CPU cores and ~350 TB DRAM, cutting hardware cost by over 90% and enabling minute‑level model and vector updates.

Overall, HELMSMAN demonstrates that when vector scales reach tens of billions, a full‑flash, clustering‑based ANNS architecture with a custom storage stack and learned pruning can simultaneously meet online SLA, drastically reduce cost, and support frequent index rebuilds, offering a practical engineering paradigm for next‑generation vector retrieval systems.