Why a High Similarity Score Still Means RAG Should Not Answer

Why a High Similarity Score Still Means RAG Should Not Answer

During an interview the interviewer asked why a RAG system should refuse to answer even when the similarity score exceeds the threshold. The author demonstrates that simply setting a similarity cutoff is insufficient.

How a Passed‑Score Answer Can Be Wrong

To illustrate the problem a minimal reproducible example is built. The knowledge base contains three insurance clauses. The retriever scores a clause by counting how many characters from the question appear in the clause; the threshold is set to 3. Two similar questions are compared:

Question 1: “Is medical expense for accidental injury reimbursed?” – the relevant clause receives a score of 8, passes the threshold, and the system answers “The policy covers medical expenses caused by accidental injury [1]”. Both required conditions (accident = accidental injury, expense = medical) are covered, so the answer is valid.

Question 2: “Can medical expense caused by sudden death be compensated?” – clause 2 scores 7 (above the threshold) and the system still returns a “compensated” answer, but none of the three clauses state that sudden death counts as an accidental injury. The necessary condition is missing, so the answer is invalid.

A third question, “Can cancer be compensated?” receives a maximum score of 1, below the threshold, and is correctly rejected. The failure case shows that a high score can hide a missing key condition.

What Conditions Are Required and What the Evidence Actually Covers

The erroneous answer is not due to a broken retriever or fabricated citations; the similarity score faithfully reflects character overlap, but the system never asks whether the retrieved evidence satisfies the question’s necessary conditions. The author lists the two judgments needed for the second question and shows that only one is covered by clause 2:

Whether medical expenses caused by accidental injury are covered – covered

Whether sudden death counts as an “accidental injury” – not covered

Because the second judgment is missing, the answer should be rejected despite the high score.

Adjusting the Threshold Cannot Fix Missing Conditions

When faced with such failure samples the first instinct is to tweak the threshold. Lowering it to 0 lets even unrelated clauses surface, producing nonsensical answers (e.g., for the cancer question). Raising it to 8 separates the two examples in this toy case, but the separation is accidental – it does not guarantee that all required conditions are satisfied.

Condition Checklist Determines Answerability First

The recommended workflow replaces the raw score check with a checklist of required conditions. For each question, list the necessary judgments, then verify whether the retrieved evidence covers each one. If any judgment is uncovered, record the gap and decide whether to ask a follow‑up, reject, or provide a partial answer.

Two implementation routes are discussed:

Manual checklist: suitable for high‑value, stable question types (e.g., insurance claims) where conditions such as accident type, expense type, and exclusions are stable.

Automated parsing: adds a step that extracts the condition list from the question but does not generate the answer itself; missing evidence stays in the “missing list”.

The extra parsing step incurs real cost (additional call, logic, occasional manual rule creation), so it should be applied to questions where a wrong answer has high audit cost.

When a failure sample appears, the practitioner should first write down the required judgments, compare them against the retrieved clauses, log any missing items, and then decide whether to improve retrieval, enrich the knowledge base, or adjust the response strategy. Global threshold changes are a last resort because they affect all queries.

Related RAG project: https://www.wushixiongai.com/projects/rag-system Threshold decides which candidates enter; evidence coverage decides whether an answer can leave.