Why Embedding Choice Outweighs Reranker in RAG Model Selection

The Limits of Public Benchmarks

Public leaderboards like MTEB help narrow the search space but cannot sign off on your business use case. MTEB covers retrieval, clustering, classification, and semantic similarity; its conclusion is that no single text representation method dominates all tasks. A model leading on multilingual average scores does not guarantee it can distinguish effective dates, regions, product versions, and exception clauses in your domain.

Current candidates differ widely. BGE-M3 offers dense, sparse, and multi-vector retrieval with 1024-dimension vectors and 8192-token max input. Qwen3 Embedding provides 0.6B, 4B, and 8B sizes, adjustable representation dimensions, task instructions, and 32K input; matching rerankers exist for each size. These specifications only indicate what can be tested — long input does not mean whole long documents should be indexed directly, and adjustable dimensions do not guarantee quality at the smallest size. Model-card fields (context length, dimensions, languages, license) serve for initial candidate filtering; final choice must come from a consistent business experiment.

Create a candidate card recording: model version, license, max input, output dimensions, instruction requirement, inference precision, deployment method, VRAM usage, batch throughput, and cold-start latency. Any hard-constraint violation removes the model before quality testing. Also lock down the calling convention: some models require task instructions for queries but not documents; some services auto-truncate or normalize. Comparing model A called via its official template against model B called with a raw encode yields meaningless results. For API models, log interface version and returned dimensions; for local models, log weight version, inference framework, and precision. Changing these conditions can make the "same" model behave differently.

What Recall and Reranking Each Solve

Embedding encodes queries and documents separately, then uses vector distance to quickly fetch candidates from a large corpus. Its primary job is not to miss correct evidence, so measure Recall@K: did the annotated relevant segment enter the top-K candidates?

Reranker receives query-candidate pairs, computes relevance scores per pair, and reorders candidates. It cares whether correct evidence is pushed to the front; metrics include MRR, nDCG@K, Precision@K, and final evidence coverage fed to the generator.

Therefore, do not only compare final answer quality with and without a reranker. If the correct segment never entered the candidate set, the failure belongs to the recall ceiling; if it entered but stayed ranked low, that is a reranking problem. Mixing both stages leads teams to spend budget on the wrong component.

Do not assume a matched pair from the same vendor is optimal. The retriever outputs candidate IDs; the reranker re-reads query and candidate text to score — they need not share a vector space. Same-series combos can be candidates, but cross-experiments (A retriever + A reranker, A retriever + B reranker) under identical candidate pool size and label set are required for fair comparison.

A common misconception: a reranker is not a knowledge completer. It can only reorder seen candidates; it cannot conjure evidence lost during chunking, permission filtering, or the recall stage.

How to Test on the Same Hard Cases

Freeze the corpus version, parsing method, chunking strategy, permission filters, and test queries. Change only one model variable at a time; otherwise you cannot attribute improvement to the model versus the data pipeline.

Build an evaluation set beyond easy keyword matches. Example query: "After out-of-area medical treatment, can outpatient expenses be reimbursed?" Positive example specifies region, visit type, and effective version. Hard negatives share keywords like "out-of-area" and "reimbursement" but describe inpatient costs, outdated policies, or a different region. Such near-miss negatives reveal whether the model matches topics or identifies answer-determining conditions.

Round 1: Test recall only. All candidate models use the same index input, same K, same filters. Record Recall@K and break down by short queries, long queries, terminology, cross-language, numeric, and version-condition categories. When overall averages are close, the failure distribution often carries more decision value than the mean.

Round 2: Fix the candidate pool, test rerankers. Every reranker sees identical queries and candidates. Compare correct-evidence ranking, hard-negative misplacement, and batch inference latency. Swapping the retriever and then the candidate pool makes it impossible to isolate the source of differences.

Round 3: End-to-end check. Freeze the generator and prompt. Verify that every key claim in the answer has supporting candidate evidence, and that the system can refuse when evidence is insufficient. High embedding and reranker scores do not automatically guarantee trustworthy final answers; this step cannot be skipped.

Labels also need auditing. A query may have multiple relevant segments with graded relevance: "direct answer", "supplementary condition", "same topic only". If only one positive label exists, a model retrieving another equally valid passage is falsely penalized. Annotators should record key conditions and evidence levels; disputed samples keep review results. Hold out a portion of the eval set with new documents, new terminology, and new versions — do not iterate on the same batch until it hits perfect scores.

After offline comparison, do not immediately switch all traffic. Shadow-run old and new pipelines on the same requests, comparing candidate differences, timeouts, truncation, and resource peaks, but let only the old pipeline serve users. Confirm index versioning, cache keys, and rollback paths work before gradually shifting traffic. This step validates whether the selection conclusion can safely enter production, independent of model intelligence.

Engineering Accounts That Make or Break Deployment

Latency account: Measure query encoding, vector search, candidate text fetch, and rerank latency percentiles separately — not a single local average. As candidate count K grows, recall quality may rise but rerank cost grows with it; K must be set under real business load.

Resource account: Parameter count affects inference compute; vector dimension affects index storage, network transfer, and search memory. Dimensionality reduction is a knob — the actual compressed dimension must be re-evaluated on the business set; do not reuse quality conclusions from the original dimension.

Length account: Derive from real query and chunk length distributions. Build dedicated truncation test cases. A model card claiming long-input support does not guarantee titles, table headers, and exception clauses in long texts are correctly utilized; over-long chunks also inflate rerank cost.

Change account: Swapping the embedding model invalidates old document vectors against new query vectors — typically requiring a new index and versioned cutover. Swapping the reranker avoids re-embedding but demands re-calibration of candidate count, thresholds, and end-to-end regression suites. Every evaluation run must bind corpus, chunking, model, and hardware versions.

Failure-cost account: Rehearse degradation scenarios during selection: Does the hosted API have a clear fallback when rate-limited? Does the local service reject new requests gracefully on OOM? Can the reranker timeout fall back to initial ranking, and does the evidence threshold still hold after fallback? A combo with high average quality that silently bypasses evidence checks during faults is not production-ready.

The most robust interview answer provides a reproducible decision chain — naming a single model is far from enough:

Hard constraints filter candidates → business hard cases test recall → fixed candidates test rerank → end-to-end validates answers → latency, resource, length, and change costs drive final trade-off.

Model leaderboards will change; this selection chain remains valid across model generations.