RAG Reranking Misses Evidence? Debug the Pipeline, Not Just the Model
When RAG reranking fails to surface known evidence, the root cause is often earlier in the pipeline: the target chunk never entered the candidate list (TopN), was truncated by token limits, or was dropped by final TopK selection—not the reranker model itself.
Reranker Cannot Retrieve What Never Entered the Candidate List
The article opens with a concrete scenario: a user asks whether nighttime taxi reimbursement requires prior approval. The knowledge base contains the exact sentence "nighttime taxi reimbursement must obtain supervisor approval first," yet the RAG answer omits this requirement. Many engineers instinctively blame the reranker model and consider swapping in a stronger one. However, the reranker only reorders candidates that have already been retrieved; it cannot invent a missing chunk.
A synthetic reimbursement corpus illustrates the pipeline. Coarse retrieval returns six chunks (A–F) in fixed relevance order. Chunk D contains the critical approval rule at its tail. With TopN=3, only A, B, and C enter reranking—D is excluded regardless of reranker quality. This demonstrates that "knowledge base has the answer" and "this request recalled the answer" are distinct; indexing, metadata filtering, chunking, retrieval model, and TopN all gate whether the target chunk reaches the reranker.
First debugging step: trace the target chunk's ID forward—was it indexed, did it pass metadata filters, what was its coarse retrieval rank, and did it enter TopN? If D never entered, fix retrieval, filtering, or candidate count; swapping the reranker solves nothing.
Chunk Entered Candidates, But Was the Key Sentence Actually Visible to the Model?
Increasing TopN to 4 brings D into the candidate list. Logs show D's ID, source page, and full original text (103 characters). However, the reranker scores a constructed pairwise input: "question + separator + candidate body" subject to a token budget. In the example, question and separator consume 17 characters, total budget is 64, leaving 47 characters for the body. The approval sentence lies at character positions [87,103) —outside the window. The reranker sees only D's header, not the decisive sentence.
Raising the budget to 256 characters expands D's pairwise input to 120 characters, finally including the approval sentence. This change only proves the model now has a chance to see the key sentence; it does not guarantee D will be ranked first.
Three diagnostic states summarize the breakpoints:
State: TopN=3 — D in candidates? No. Approval sentence in actual input? No. Action: fix retrieval entry first.
State: TopN=4, budget 64 — D in candidates? Yes. Approval sentence in actual input? No. Action: fix input truncation.
State: TopN=4, budget 256 — D in candidates? Yes. Approval sentence in actual input? Yes. Action: now evaluate ranking quality.
Real models count tokens, not characters. Token allocation for question, candidate, special tokens, and truncation strategy (truncate longer side vs. only second segment) depend on model, wrapper, and version. Debugging must save or reconstruct the exact tokenized input sent to the model; pre-truncation original text is not a substitute.
Source metadata (page number, section title) logged alongside the chunk does not automatically become part of the pairwise text. If those fields are not concatenated into the input, the reranker never sees them. Similarly, the question containing the word "approval" does not count as the source evidence being visible.
Multi-Evidence Questions: What Does Recall Rate Actually Measure?
For a single-fact question ("Does nighttime taxi need approval?"), one evidence unit (the rule in D) suffices. D entering candidates and its key sentence being visible counts as coverage.
Change the question to "What are the approval and attachment requirements for cross-city nighttime taxi?" Now two distinct evidence units are required: D's supervisor approval and F's "cross-city nighttime taxi must attach itinerary description." With TopN=4, only D is present; F (rank 6) is absent. Reporting "reimbursement policy recalled" at document level masks the fact that only one of two required facts was retrieved. Evidence units must align with task requirements: overlapping chunks from the same rule count as one unit; different rules from the same document count as separate units.
Candidate coverage and visible coverage must be tracked separately. D may be in TopN (candidate coverage hit) but its key sentence truncated (visible coverage miss). Recording both distinguishes "chunk not retrieved" from "chunk retrieved but key content not passed to model."
Final TopK imposes another ceiling. The cross-city question needs both D and F; if TopK=1, only one can be kept even if both are fully visible. If a single chunk happened to contain both rules, the conclusion would differ—so "two facts" does not mechanically imply "must keep two chunks."
Even with all necessary evidence in context, the generator must still understand conditions, combine information, and produce a correct answer. Candidate coverage, input visibility, final retention, and answer correctness are four adjacent but distinct checkpoints.
Cost-Benefit of Increasing TopN
Since D is at rank 4 and F at rank 6, raising TopN seems an obvious fix. However, each additional candidate adds one pairwise input for the reranker. Testing N=3,4,6 yields 6, 8, 12 pairwise inputs respectively.
In this fixed list: N=3 → approval question 0/1, cross-city 0/2 N=4 → approval 1/1, cross-city 1/2 (E adds no value) N=5 → same as N=4 (E only adds compute) N=6 → cross-city reaches 2/2 (F finally included)
The benefit of increasing N must be measured by how many required evidence units are newly covered, not by raw candidate count. The 5th candidate only increases reranking workload without helping either question; the 6th finally supplies the attachment requirement.
Pairwise input count is the easiest cost proxy. Real deployment costs also depend on token length distribution, batch size, device utilization, queue time, and throughput. Ten short candidates differ from ten near-limit candidates. Batching may improve overall throughput but can increase latency to fill batches; "supports batch" does not equal "single request faster."
Select N by scanning a few values on a validation set, recording both evidence coverage gain and actual compute/latency/resource increase. Finalize parameters, then report on held-out questions to avoid overfitting to the tuning set.
Tracing Evidence Disappearance by Stable Candidate ID
If reranking failures only log final scores, later diagnosis is hard. A practical approach: assign stable IDs to candidates and chain these records: coarse retrieval rank, document source, pre-truncation original text, actual pairwise input, rerank score, rerank rank, and final retention flag.
For chunk D:
Absent from coarse retrieval list → problem in indexing, chunking, metadata, or filtering.
Present in list but missing from TopN → retrieval quality or candidate count.
In TopN but key sentence absent from pairwise input → chunking, concatenation template, or token budget/truncation strategy.
Key sentence visible but score low → model fit, training data.
Ranked high but missing from final context → threshold, deduplication, TopK, context budget.
The following mapping summarizes the first missing position and priority checks:
Before coarse retrieval : Indexing, chunking, metadata, filter conditions
Outside TopN : Retrieval quality, candidate count
In pairwise input : Concatenation template, token budget, truncation strategy
In rerank results : Model scores, input order, ID alignment
In final context : Threshold, deduplication, TopK, context budget
During batch scoring, verify that scores align with candidate IDs. If input order changes but output scores are mapped back to the old order, a false "model misranking" illusion appears. Saving a detached score list is insufficient; input ID sequence and output ID sequence must be cross-referenceable.
This localization method ensures each failure mode maps to a specific repair layer. It avoids blaming the reranker for every failure and prevents prematurely declaring evidence complete just because a document was recalled.
Interview Answer: State the Ceiling and the Verification Steps
When asked "RAG reranking misses evidence, what do you do?", structure the response as follows:
I would not start by swapping models. Instead, I trace the retrieval pipeline to locate where the target evidence first disappears. The reranker only sorts the provided TopN candidates, so first verify the target chunk was indexed, passed filters, and entered the candidate set.
After candidate hit, inspect the actual pairwise input fed to the model to confirm the key sentence was not lost to chunking or token budget truncation.
For multi-fact questions, define evidence units per question, then separately log candidate coverage, visible coverage, and final TopK retention. When increasing TopN, compare newly covered evidence against added pairwise inputs, token consumption, and latency—not just recall numbers.
Only when the target evidence is fully visible within a fixed candidate set, candidate IDs align with scores, yet it still ranks poorly, do I compare reranker models under identical input conditions. That model swap then addresses ranking capability, not upstream missing evidence.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
