Does Reducing Embedding Dimensions Break RAG Retrieval? Practical Trade‑offs and Tests

Answering the "Does reduction crash?" question

First define what "crash" means: a drop of Recall@100 by 5% or NDCG@5 by 20%, either on the overall mean or on specific query groups.

Vector reduction is information compression; loss is inevitable. Techniques such as MRL or PCA aim to make the loss controllable.

Scenario considerations

Task sensitivity : Fine‑grained domains (legal, medical) are more vulnerable than simple FAQ retrieval.

Scale factors : Document volume, QPS, storage and ANN index cost determine whether the loss is acceptable.

Overall trade‑off : Latency, memory usage and retrieval metrics must be weighed together.

MRL (Matryoshka Representation Learning)

During training, prefixes of the embedding (e.g., first 64, 128, 256 dimensions) are forced to be independently searchable, similar to a Russian‑doll structure.

Loss : Standard models compute loss on the full vector; MRL computes loss on each truncation length and aggregates them.

Information distribution : Standard models spread information uniformly across dimensions; MRL concentrates important signals in early dimensions.

Inference truncation : Standard models cannot be safely truncated; MRL can be truncated at any trained length with relatively controlled loss.

Conclusions on MRL

MRL is not loss‑less; it merely encodes which dimensions are more valuable.

Truncation should use lengths seen during training (e.g., 1024/512/256/128/64). Unseen lengths have no guarantees.

MRL is a training method, independent of the absolute dimensionality (768/1024/3072, etc.).

Whether a model supports MRL must be verified from the paper or vendor docs, not inferred from dimension count.

Non‑MRL dimensionality reduction

Option A: PCA

Procedure: run embeddings on a sample of business documents, compute PCA on the embedding matrix, keep the top k principal components, and project new vectors as new_vector = old_vector × projection_matrix (same matrix for query and document).

Pros : Unsupervised, quick to implement, low cost.

Cons : Linear transformation; optimises variance, not retrieval metrics; requires a representative sample, otherwise the projection may be biased.

Option B: Supervised projection layer

Procedure: freeze the original embedding, add a linear or small MLP layer, train it with contrastive learning on query–document relevance pairs (labels or click logs).

Pros : Directly aligns with the retrieval task; non‑linear capacity.

Cons : Needs labeled or reliable weak signals; risk of over‑fitting and distribution shift; higher engineering cost.

Choosing between them

First evaluate PCA on a validation set using Recall@K / NDCG@K. If it meets requirements, stop. If not and you have annotations, consider the supervised projection.

PCA does not require relevance labels but needs a representative document sample.

Evaluation metrics

Recall@K = (relevant docs in top K) / (total relevant docs for the query)

– measures missed relevant docs, independent of rank order. Precision@K = (relevant docs in top K) / K – measures purity of the returned set. F1 = 2 × (Precision × Recall) / (Precision + Recall) – harmonic mean, penalises extreme imbalance. DCG@K = Σ rel_i / log₂(i+1) – raw gain, ignores position discount. IDCG@K is the ideal DCG (all relevant docs sorted by score). NDCG@K = DCG@K / IDCG@K normalises to 0‑1 for cross‑query comparison.

Typical practice: use large K (e.g., 50, 100) for recall, small K (1‑5) for precision/NDCG. Report per‑query‑type averages when the number of relevant docs varies widely.

Illustrative calculation

System returns top‑3 scores: 1, 3, 2; true relevance scores: 3, 2, 1.

DCG@3 = 1/log₂2 + 3/log₂3 + 2/log₂4 ≈ 1 + 1.893 + 1 = 3.893

IDCG@3 (ideal order 3, 2, 1) ≈ 3 + 1.262 + 0.5 = 4.762

NDCG@3 ≈ 3.893 / 4.762 = 0.818

Common pitfalls

Recall@100 may stay flat while NDCG@5 drops sharply – the relevant docs are still retrieved but pushed down.

High‑dimensional vectors cause Euclidean distances to concentrate; the relative variance shrinks as ∝ 1/√d, making discrimination harder for ANN indexes.

Signal vs. noise in dimensions

Adding dimensions that are pure noise raises the overall distance but dilutes the true signal, reducing the signal‑to‑noise ratio. Adding truly informative dimensions improves discrimination.

Dimensionality reduction that removes noisy dimensions can act like regularisation and improve metrics; removing dimensions that contain fine‑grained signal will hurt performance.

Long‑tail queries

Long‑tail refers to low‑frequency intents, not sentence length. Such queries have few training examples, so their signal is weak and more likely to be lost after reduction or with weaker models.

Decision rule: weigh frequency × per‑error cost . Low‑frequency, low‑cost cases can tolerate degradation; low‑frequency, high‑cost domains (legal, medical, finance) require protection.

Diagnosing what was cut

Do A/B tests on your own qrels per query group (high‑freq vs long‑tail, with/without negation, etc.). Look for uniform small drops (overall loss) versus sharp drops in specific groups (e.g., long‑tail or terminology).

The worst cases are often the intersection of true signal, long‑tail intent, and originally weak representation – they disappear first when truncating.

Fundamental limits of vector retrieval

Vector search is approximate semantic similarity, not exact keyword matching. Increasing dimensionality alone cannot achieve exact match.

Production systems commonly use hybrid retrieval (dense + sparse/BM25) to compensate for dense‑only limitations.

RAG optimisation ceiling

Rerankers (cross‑encoders) only reorder the candidate pool returned by the dense retriever; they cannot recover documents that never entered the pool.

Techniques that can affect the pool include query rewriting/HyDE (pre‑retrieval), hybrid retrieval, and late‑interaction models like ColBERT.

Scaling costs

ColBERT (hundreds of millions of docs) : storage grows roughly as token‑count × dimension, often two orders of magnitude larger than single‑vector embeddings; token‑level interaction is computationally expensive. Typical deployment runs ANN for coarse recall, then applies ColBERT only on the top‑N candidates.

HyDE (tens of millions of QPS) : each query triggers an extra LLM generation, linearly increasing latency and cost. Common practice is to enable it only for difficult queries, with caching and smaller models.

Key take‑aways

Optimisations that only reorder inside the candidate pool are limited by the recall of the initial dense retrieval.

Changing the query, using hybrid retrieval, or multi‑vector approaches can improve recall, but each brings storage, latency, and cost constraints that require hierarchical, cached, and selective activation at scale.