Understanding RAG: Chunking, Embedding, Similarity, HNSW and Multi‑Path Retrieval

Why RAG Is Needed – The Limits of LLMs

LLMs often hallucinate, have a fixed knowledge cutoff, and cannot access private enterprise data, leading to incorrect or fabricated answers when asked about internal systems or code. Retrieval‑Augmented Generation (RAG) solves this by first retrieving relevant external information and then feeding it to the LLM.

RAG Pipeline Overview

RAG consists of two stages:

Document → Chunking → Embedding → Index write (vector DB)

and

User query → Embedding → Recall (ANN + BM25) → RRF fusion → Rerank → LLM generates answer

The offline stage pre‑processes documents into chunks and stores their vectors; the online stage transforms the query, retrieves candidates, fuses rankings, reranks, and finally generates a response.

Knowledge Sources

Static shared knowledge includes product docs, code repositories, chat logs, and HR policies. Dynamic personal memory captures user preferences, recent behavior, and context‑dependent information (e.g., a user consistently asks for Go + Gin code).

Embedding: Turning Text into Computable Data

Embedding models map a piece of text to a high‑dimensional vector (commonly 768, 1024 or 1536 dimensions). The model is trained by contrastive learning to bring semantically similar sentences close together and push unrelated ones apart.

Example of a 2‑D illustration:

"用户喜欢川菜" → [0.21, -0.43, 0.07, 0.85, ..., -0.12]  ← 1536 numbers

Dense vectors capture meaning but are insensitive to exact tokens; sparse vectors retain term‑level information, making the two representations complementary.

Chunking – How to Split Knowledge

Chunking is critical because feeding an entire document to the embedding model either exceeds token limits or dilutes semantic relevance. Three main strategies are discussed:

Fixed length + sliding window: e.g., 500‑token chunks with 200‑token overlap to avoid cutting sentences.

Semantic chunking: split by natural document structure (headings, paragraphs).

Parent‑Child chunking: large parent chunks preserve context while smaller child chunks are used for retrieval; retrieved children are re‑assembled under their parent before passing to the LLM.

Similarity Measures

The most common metric is cosine similarity: cos(θ) = (A · B) / (|A| × |B|) Values range from –1 to 1; in practice embeddings produce scores in [0, 1]. Euclidean distance and dot product are also mentioned, but cosine is recommended because embedding models are trained to optimize it.

Concrete example:

A = [3, 4]   // "今天天气真好"
B = [3.5, 3.8] // "外面阳光明媚"
cos(A,B) ≈ 0.99  // highly similar
C = [-1, 0]    // "电脑坏了怎么办"
cos(A,C) = -0.6 // unrelated

HNSW – Fast Approximate K‑Nearest Neighbor Search

Exact KNN scales poorly (O(N·D)). HNSW combines a skip‑list‑like hierarchy with a navigable small‑world graph to achieve logarithmic search complexity.

Key steps:

Randomly assign each node a level using P(level ≥ L) = (1/M)^L (M≈16).

During insertion, perform greedy search from the top layer to find nearest neighbors.

Connect the new node to its M nearest neighbors on every layer it appears (bidirectional edges).

Prune excess edges to keep at most M per node.

Search proceeds layer‑by‑layer: greedy descent on each layer followed by a final fine‑grained search on layer 0.

Typical parameters: M (connectivity), ef_construction (index build trade‑off), ef_search (query‑time recall vs speed).

Multi‑Path Recall and Metadata Filtering

Before recall, a metadata filter narrows the candidate set (e.g., tenant_id, document type). Recall can combine several paths:

Recall
├─ ANN (vector similarity)
├─ BM25 (keyword inverted index)
├─ Graph Recall
├─ SQL Recall
├─ Memory Recall
├─ Conversation Recall
├─ Web Search
└─ Tool Recall

Because scores from different paths are incomparable, the article introduces Reciprocal Rank Fusion (RRF) to merge rankings.

RRF – Rank‑Based Fusion

RRF discards raw scores and sums the reciprocal of ranks with a smoothing constant k (commonly 60): score(doc) = Σ 1 / (k + rank_i) Example with two paths (ANN and BM25):

ANN ranking: [doc_A #1, doc_B #2, doc_C #3]
BM25 ranking: [doc_B #1, doc_C #2, doc_A #3]

doc_A score = 1/(60+1) + 1/(60+3) = 0.0323
doc_B score = 1/(60+2) + 1/(60+1) = 0.0325
doc_C score = 1/(60+3) + 1/(60+2) = 0.0320

Resulting order: doc_B > doc_A > doc_C.

Rerank – Cross‑Encoder Re‑Scoring

Recall (especially Bi‑Encoder embeddings) can retrieve semantically related documents but may rank them incorrectly. A cross‑encoder processes the query [SEP] document pair jointly, producing a relevance score that is directly comparable across candidates.

Example:

Bi‑Encoder scores: VPN password reset (0.85), VPN client install (0.83)
Cross‑Encoder scores: VPN password reset (0.98), VPN client install (0.31)

After reranking, the truly relevant document moves to the top.

Full End‑to‑End Flow

Query Rewrite → Metadata Filter → Recall (ANN + BM25) → RRF Fusion → Top‑N → Rerank (Cross‑Encoder) → Top‑K → LLM generates answer

Key takeaways highlighted in the article:

Chunking strategy determines what knowledge can be retrieved.

Embedding converts meaning into vectors; dense captures semantics, sparse captures exact terms.

Cosine similarity is the default metric for embeddings.

HNSW provides fast ANN search for millions of vectors.

Query rewrite and metadata filtering are high‑impact optimizations.

Multi‑path recall with RRF fusion and cross‑encoder rerank yields the best overall performance.