RAG Retrieval: Comparing Bi-encoder and Cross-encoder Architectures

RAG Retrieval Stages

RAG consists of three stages: Retrieve – find the most relevant document fragments from an external knowledge base; Augment – inject the retrieved content into the LLM context; Generate – let the LLM produce an answer based on the augmented context. Retrieval quality directly determines overall system performance.

Semantic matching is the dominant retrieval method and must satisfy two requirements: speed (millisecond‑level latency) and accuracy (deep language understanding beyond bag‑of‑words).

Bi‑encoder

Architecture

A dual‑tower Bi‑encoder contains an independent Query Encoder and Document Encoder. The two encoders operate without any token‑level cross‑attention; matching is performed solely by similarity in the vector space (cosine, inner product, Euclidean).

Advantages

Document vectors can be pre‑computed offline, enabling constant‑time online latency regardless of knowledge‑base size.

Offline stage: all documents → DocEncoder → document vectors → vector index (FAISS, HNSW, IVF‑PQ, DiskANN)

Online stage: query → QueryEncoder → query vector → ANN search → top‑K documents

Limitations

The entire semantic content of a query and a document must be compressed into a single fixed‑dimensional vector, causing loss of fine‑grained signals such as word sense, negation, numeric matching, or long‑range dependencies. Bi‑encoders are therefore best suited for coarse‑grained filtering.

Cross‑encoder

Architecture

Cross‑encoders concatenate the query and document into a single sequence and feed it to a Transformer. Every token of the query can attend to every token of the document at each self‑attention layer (all‑to‑all attention).

Advantages

Precise disambiguation (e.g., “Apple” the company vs. the fruit).

Word‑order and syntactic awareness (e.g., “A acquired B” vs. “B acquired A”).

Negation handling.

Long‑distance dependencies because query tokens can directly attend to any document token.

Limitations

Document representations cannot be pre‑computed; each query‑document pair requires a full forward pass. For a knowledge base of 1 million documents, a BERT‑base Cross‑encoder (~300 M parameters) would need ~1 million forward passes (hours), and a 7 B LLM would need 100–500 ms per pass, making exhaustive retrieval infeasible. Consequently, Cross‑encoders are typically used only as re‑rankers.

Hybrid Retrieval (Bi‑encoder + Cross‑encoder)

Typical Parameters

K (recall size): 100 – 1000 (larger K improves coverage but increases re‑ranking latency).

N (documents injected into LLM context): 3 – 10 (limited by LLM context window).

Vector dimension: 256 – 7168 (higher dimension improves representation power but raises storage and retrieval cost).

ANN index type: HNSW, IVF‑PQ, DiskANN (trade‑off between recall, speed, and memory usage).

Engineering Optimizations

Reduce K to lower Cross‑encoder calls.

Use a distilled lightweight Cross‑encoder.

Apply ONNX quantization or TensorRT acceleration.

ColBERT (Late Interaction)

ColBERT (Contextualized Late Interaction over BERT) combines Bi‑encoder offline indexing with token‑level interaction at match time.

Core Idea

During encoding each token retains its own vector (e.g., 128‑dim). At matching, each query token finds its most similar document token (MaxSim) and the scores are summed.

Advantages

Token‑level vectors can be pre‑computed and indexed.

Matching preserves token‑level interaction, improving accuracy over single‑vector Bi‑encoders.

Retrieval quality approaches that of Cross‑encoders while remaining much faster.

Storage Cost

Storage ≈ average_token_count × (per_token_dim / dense_dim). Example: 128‑dim token vectors with 200 tokens per document → about 200 × the storage of a single‑vector approach.

ColBERTv2

ColBERTv2 compresses token vectors via residual quantization, reducing storage by an order of magnitude and adding denoising supervision for more stable training, achieving production‑grade latency while retaining late‑interaction accuracy.

LLM as Re‑ranker

Large language models can replace or augment traditional Cross‑encoders for re‑ranking. Although not dramatically cheaper or faster, they provide stronger zero‑shot, temporal generalization and instruction‑following capabilities, making them suitable for complex, evolving ranking tasks.

Conclusion

The classic efficiency‑accuracy trade‑off appears in semantic retrieval: Bi‑encoders provide speed for coarse‑grained recall, while Cross‑encoders deliver precision for fine‑grained re‑ranking. In practice, start with a simple Bi‑encoder + vector database, then iteratively introduce a lightweight re‑ranker (e.g., distilled Cross‑encoder, specialized LLM, or ColBERT) based on data‑driven evaluation of recall, latency, and resource constraints. For extremely high‑concurrency or strict deterministic latency scenarios, dedicated Cross‑encoders or multi‑vector indexes such as ColBERT may still be appropriate. Each upgrade should be validated through systematic offline testing and online metrics to avoid premature complexity. Reference: https://arxiv.org/pdf/2203.08372