Why Tweaking Prompts or Top‑K Won’t Fix RAG Errors – Trace the Evidence Layer

Answer Within 30 Seconds

I will trace the evidence layer by layer: first verify that the correct document, page, or chunk was parsed and indexed; then compare the original Query with the rewritten Query for negations, entities, and filters; next inspect BM25, vector, fusion, and rerank candidates, scores, and ranks; finally confirm that the model’s context contains the full evidence. Record Chunk ID, source, retrieval channel, filter conditions, model and index versions at each step, and validate with the original Query and the same evidence on regression.

Identifying the correct evidence is the anchor for any debugging; without it there is no reference point.

Why Start with Correct Evidence?

A correct piece of source text is the coordinate of the entire debugging chain. A reproducible sample must include the original Query, the reference answer, and the exact location of the correct evidence (document, page, chunk, or content region).

Layer 1 – Was the Correct Content Indexed?

Check offline ingestion: did the source file reach the system, was it parsed correctly, and were the resulting chunks written to the searchable index? Verify that the chunk exists in the current index version and that no batch write failures or version mismatches prevented it from being searchable.

Layer 2 – Was the Query Processed Incorrectly?

Ensure the original Query and any rewritten Query are saved. Examine whether negations, entities, or filters were lost, and whether the query was improperly expanded or split, especially for multi‑hop questions.

Layer 3 – Did Filtering Remove the Correct Evidence?

Filters (knowledge‑base ID, tenant, permissions, document status, version, time, type) are applied before relevance ranking. Record the actual filters used at request time and test whether the correct chunk appears without filters, then re‑apply them one by one to locate the offending rule.

Layer 4 – What Did Each Retrieval Path Return?

Separate BM25 and vector retrieval results. BM25 favors exact terms; vector search handles semantic rewrites. Save candidates, scores, and ranks for each path to see which one originally retrieved the evidence and whether fusion later suppressed it.

Layer 5 – Why Did Fusion Push the Evidence Down?

Fusion combines heterogeneous scores; verify normalization, weighting, and any rank‑based merging logic. Ensure stable IDs are used for deduplication so that a correctly retrieved chunk isn’t inadvertently dropped.

Layer 6 – Did Rerank Have a Chance?

Rerank can only reorder existing candidates. If the correct evidence never entered the top‑N, changing the rerank model won’t help. Compare pre‑ and post‑rerank ranks and check for truncation or model‑input mismatches.

Layer 7 – Is the Final Context Complete?

After ranking, the selected chunks are assembled into the model’s context. Verify that the evidence wasn’t truncated, that multi‑hop questions have all required chunks, and that no duplicate or noisy chunks consumed the context window.

Freeze the Experiment Environment

Capture a minimal reproducible package: original Query, history, filter conditions, reference answer, correct evidence location, BM25/vector/fusion/rerank top candidates, and versions of parsers, index, embeddings, rerank, and prompts. This enables exact replay of the failure.

Use a Candidate‑Change Table

Record for each layer whether the evidence exists, its rank, and why it was kept or filtered. This table reveals where the evidence vanished and guides the appropriate fix.

Avoid One‑Off Fixes

After fixing, retest with the original Query and the same evidence, add adjacent samples of the same failure type, and change only one variable at a time. Verify both retrieval improvements (evidence rank) and end‑to‑end answer correctness.

Overall, the key to solving RAG failures is systematic evidence tracing rather than blind parameter tuning.