Deep Dive into SINDI: High‑Performance Sparse Vector Search Index for VSAG

Why SINDI Is Needed

In Retrieval‑Augmented Generation (RAG), intelligent search and recommendation systems face mixed user queries that combine exact terms, synonyms, and vague intents. Dense retrieval excels at semantic generalisation but struggles with fine‑grained tokens, while traditional BM25 excels at keyword matching but lacks semantic depth. Combining BM25, sparse vectors, and dense vectors yields the best recall, and SINDI is built to provide a high‑performance, low‑cost sparse‑vector retrieval component.

Sparse Vectors Overview

A sparse vector records only non‑zero term‑weight pairs from a very high‑dimensional vocabulary (tens of thousands of dimensions). Each non‑zero dimension corresponds to an interpretable term, enabling both explainability and efficient storage as {term: weight} pairs rather than full dense arrays.

Example: the sentence "I love black cat" can be represented as a dense vector [0.18, -0.42, 0.07, ...] (hard to interpret) or as a sparse vector {love:0.71, black:0.82, cat:0.93, feline:0.51, kitty:0.28}, where each entry maps directly to a term.

Core Value of SINDI

SINDI was published at ICDE (CCF‑A) and targets sparse‑vector retrieval. Compared with a traditional inverted index, SINDI builds a 1 000 000‑document Chinese dataset in 33.0 seconds with an index size of 4.99 GB , versus 551.4 seconds and 19.18 GB for the baseline. Open‑source graph indexes take up to 1915.77 seconds . In the BigANN sparse track, SINDI’s QPS remains optimal across recall levels (90 % → 99 %).

Key Design of SINDI

2.1 Window‑Based Weighted Inverted Structure

SINDI stores a Value‑based Inverted List where each term list keeps both document IDs and the term’s weight. Documents are partitioned into windows (10 000–60 000 docs) and each window uses uint16_t IDs, reducing storage and keeping the distance table for a window cache‑friendly. Query scoring becomes a sequential read of query_weight × doc_weight summed into a per‑window distance table, avoiding random accesses to full vectors.

2.2 Term‑Based Scoring

During search, SINDI iterates over windows, then over query terms. For each posting (doc_id, weight) it adds query_weight × weight to the distance table. After processing all terms in a window, the table holds the approximate inner‑product scores for candidate docs, which are merged into a global top‑k heap.

2.3 Multi‑Layer Pruning

Three pruning layers reduce computation:

Document‑side pruning (build time) drops low‑weight terms per document based on a cumulative quality ratio (e.g., keep 80 % of total weight).

Query‑side pruning (search time) retains only the highest‑weight query terms (e.g., top 75 %).

Term‑list pruning scans only the top‑percentage of postings in a term’s list (e.g., first 70 %).

These layers can be combined gradually, guided by the analyze tool that reports recall, QPS, and index size.

2.4 Quantization & Reranking

To further cut memory, SINDI applies 8‑bit scalar quantisation (SQ8) to the inverted list values, halving memory with negligible accuracy loss. For the forward (rerank) data, Distribution‑Maintenance Quantisation (DMQ) packs term IDs and 8‑bit values, achieving ~50 % compression with only ~1 % recall drop and <10 % QPS reduction.

2.5 Index Layout

The index consists of two parts:

Inverted index : window_term_list_ → SparseTermDatacell containing doc_id[] (uint16) and value[] (float or quantised).

Forward data for reranking: per‑document blocks storing len, term_id[], and value[], with an offset table mapping doc IDs to blocks.

Using SINDI in VSAG

Example build parameters (C++):

std::string sindi_build_parameters = R"({
    "dtype": "sparse",
    "metric_type": "ip",
    "dim": 128,
    "index_param": {
        "term_id_limit": 1000000,
        "window_size": 60000,
        "doc_prune_ratio": 0.0,
        "use_quantization": false,
        "use_reorder": true,
        "remap_term_ids": false
    }
});
auto index = vsag::Factory::CreateIndex("sindi", sindi_build_parameters).value();

Search parameters example:

std::string sindi_search_parameters = R"({
    "sindi": {
        "n_candidate": 200,
        "query_prune_ratio": 0.0,
        "term_prune_ratio": 0.0,
        "use_term_lists_heap_insert": true
    }
});
auto result = index->KnnSearch(query, 10, sindi_search_parameters).value();

Conclusion

SINDI provides a high‑performance sparse‑vector inverted index that leverages windowed data layout, weighted posting lists, multi‑layer pruning, 8‑bit quantisation, and a high‑accuracy rerank stage. It excels in scenarios requiring semantic expansion with explainable term matches, large‑scale retrieval with tight memory and latency budgets, and debugging via term‑level inspection. It complements dense indexes in VSAG but is not a replacement for pure dense retrieval or tiny datasets.

Limitations include lack of hard delete support and limited filter capabilities; tuning window size, candidate count, and pruning ratios is essential for optimal performance.