Why Rebuilding RAG Indexes Fails to Fix Stale Policy Answers
This article details how to guarantee RAG systems retrieve the correct policy version after updates, covering parsing, chunking, vector generation, index switching, cache invalidation, and request tracing, using a reimbursement policy change example to illustrate versioning pitfalls and debugging strategies.
Core Problem: RAG Returns Old Policy After Update
The author illustrates a common failure: after uploading a new policy PDF (e.g., accommodation reimbursement limit raised from 500 to 600 yuan effective Nov 10), the RAG system still answers 500 yuan with seemingly complete citations. Simply saying "I rebuilt the index" does not answer the interviewer's follow-up: "Which version did this request actually query?"
Q1: How to Guarantee RAG Uses the Correct Version After Policy Update?
Build new data separately (r2), verify it, then switch the query entry point. A single request must use the same data batch from retrieval to citation, and caches must be partitioned by that batch. When an old answer appears, trace the request: which entry was chosen, which version was retrieved, which version was fed to the model, and whether an old cache was hit. A backend "update success" message only proves an operation finished; it does not testify for a specific request.
Distinguish policy versions (v1, v2) from system releases (r1, r2). An r2 release can retain expired, current, and future policies to answer questions for different dates. Switching to r2 still requires selecting the correct policy for the query.
Q2: Why Does Old Content Persist After File Upload?
The path from uploaded file to model-visible chunks has several steps:
PDF parsing : Open the parsed output and locate the clause containing 600 yuan. Check whether the new amount was misread as the old amount, whether footer notes were missed, or whether a download task fetched an old file with the same name. If the parsed text lacks 600, later retrieval tuning is useless.
Chunking : The amount and its effective date ("effective from Nov 10") may land in separate chunks. Retrieval might only find the amount chunk, causing the answer to ignore the date condition. Keep all conditions for a clause together and ensure chunks can trace back to file, version, and position.
Vector generation : Vectors are the numeric representations used for retrieval. When body text changes, affected chunks must be re-vectorized. Changing a chunk's version tag to v2 does not update its old vector; conversely, keeping old vectors with new text creates mismatch. The author compares original text, parsed text, and indexed chunks side by side to pinpoint the first step that diverges.
Q3: Can Users Query While New Index Is Half-Built?
No. Finding one new clause does not prove the rest are ready. In the example, the new version also allows electronic invoices. If only the amount chunk was indexed, "What is the limit?" works but "How to submit invoices?" fails. The author's design keeps the serving release (r1) live while building r2 elsewhere. Verification uses a clause list derived from the source document (not from already-indexed count) to check: every expected clause present, text matches the version, vectors generated from correct text. Only after passing does the query entry switch from r1 to r2.
Example clause-to-segment mapping: beijing-v2:receipt must contain the electronic-invoice rule. If real chunking splits one clause into two segments, both must be recorded; hitting one is not enough. When generating vectors, store the text fingerprint and vector algorithm version. The fingerprint detects text changes; the example's term-frequency algorithm catches "new text with old vector"; real models must preserve input-output correspondence of generation tasks.
Q4: In a Mixed-Version Corpus, Just Pick the Highest Version?
Selection must consider applicability: employee scope, business date, policy approval/effective/expired status. A Nov 9 query cannot use a Nov 10 policy; a Dec adjustment cannot answer Nov questions; another region's higher limit is irrelevant. These filters apply at retrieval time: first determine allowed documents by scope, date, and status, then search for relevant clauses. Similarity only ranks relevance; it does not judge effectiveness. If two same-date policies conflict, escalate to policy maintainers; picking the highest version may select a wrong upload. Ask clarifying questions when employee region or date is ambiguous. Historical reimbursement queries must use the business date, not today's policy.
Q5: Entry Switched to r2, Why Still Return 500 Yuan?
Cache hits can bypass retrieval. The author's local test: r1 answers 500, switch to r2 with correct filtering answers 600. But with a cache key of "daily accommodation reimbursement limit", the same question hits the old cache record, returns 500, citation still points to v1. The test uses simple term-frequency vectors and direct evidence extraction (no LLM) to isolate the cache effect. Rebuilding the index does not help because the old answer never went through the new index.
Cache keys must distinguish region, business date, release batch, and policy identity. In the example, the key includes all four, so r2 requests miss r1's cache. If the final answer is cached, model or prompt changes also require invalidation. The author shows a request trace table comparing failed and fixed runs (same question, Beijing, Nov 10):
Entry selection : both r2/v2
Answer cache : failed hits r1/v1 old entry; fixed misses
Actual retrieval : failed not executed; fixed finds v2 accommodation clause
Model context : failed not built; fixed includes 600-yuan clause
Return & citation : failed 500/v1; fixed 600/v2
The "not executed" rows indicate early cache return. Logs must record such early exits explicitly. After fix, rerun the same question: old cache misses, answer becomes 600 with v2 citation. Verifying both answer and citation prevents wording changes with stale evidence.
Q6: Troubleshooting Order for a Request Returning Old Answer
Assign a request ID; link all logs by this ID.
Check entry record and cache hit. If entry is r1, verify r2 build completion, publish success, and instance entry update. If entry is r2 but old cache hit, follow cache invalidation logic.
If no answer cache hit, inspect retrieval candidates. If wrong-version candidates appear, check the actual query-time version and date filters (not just code). If expected clause missing, trace back to parsing, chunking, indexing.
If candidates correct, examine final model context (reranking and concatenation steps). Constraints may be dropped; old chunks from other retrieval branches may mix in.
If context and citations correct but answer still wrong, investigate generation: reused prior conclusion? Prompt enforcing current evidence? Asking the model to "use latest policy" does not reveal which earlier step fetched the wrong version.
Every layer must log chunk identity and version; citations must resolve to original text. A single request ID ties the chain together; do not splice screenshots from different runs. Cache hits must also log which data batch produced them and which policy they cite. Troubleshoot by locking one failed request, then rerun with identical employee scope, business date, and question. Compare pre-fix (hits r1 cache) and post-fix (r2/v2 retrieval, 600 yuan). Different regions or questions do not prove this specific fix works. Keep original and rerun request IDs linked in fix records to avoid backfilling new evidence into the old request.
Q7: Request Mid-Flight During Version Switch
Define which moment the request should reflect. Fix the request's data batch at start so retrieval and citation stay consistent. However, policy Q&A has timeliness: if Nov 10 policy is already effective, an in-flight request that would return 500 is non-compliant even if internally consistent. If business requires current policy at response time, re-check release batch before responding; switched requests rerun on new batch, and missing applicable evidence indicates missing data. These are two separate designs: intra-request consistency and business-time compliance. Real systems must handle concurrent re-switches; a local sequential example cannot claim full concurrency safety.
Q8: How to Prove Synchronization After Fix?
Run all questions affected by the policy change, checking both answers and citations. The new limit must change for current date; historical questions stay on old rule; other regions unchanged; new invoice clause retrievable. Also test a previously cached old question to ensure it does not bypass new data. The author's local test covered five scenarios: current date, pre-effective date, other region, future date, new clause. Passing these confirms version selection logic for those inputs; retrieval quality and production reliability still need system-level testing.
Before interviews, prepare one real project policy update record: source document, indexed chunks, selected release batch, cache hit status, final citations. Explain design where it's design; show before/after logs for steps actually changed. When asked "Which version did this request query?" you can trace 500's origin and 600's arrival.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
