Why RAG Misses Casual User Questions and How to Optimize Retrieval

Why Queries Fail

In production many user questions are unsuitable for direct retrieval. Typical failure categories and their concrete examples are:

Multi‑turn coreference : “那企业版呢？” – missing previous context.

Colloquial expression : “一直转圈进不去” – document uses the term “认证超时”.

Abbreviation / alias : “SSO 咋开” – document writes “单点登录”.

Compound question : “能导出吗，失败后怎么补偿？” – two intents mixed together.

Phenomenon description : “升级后以前的数据没了” – documents are organized by cause/module, not by the vague description.

Overly verbose : “一大段背景加一个真正问题” – irrelevant words dilute the retrieval signal.

Classifying failures helps decide which optimization method to apply.

Query Rewrite: Turning Dialogue into Stand‑Alone Queries

Query Rewrite is best for handling coreference, colloquial language, typos, and missing context. It converts a conversational turn into an independent, self‑contained query that can be understood without the chat history.

Example conversation:

用户：Windows 客户端出现 1007 错误怎么处理？
助手：可以先关闭代理并清理缓存。
用户：那 Mac 呢？

Rewritten independent query:

macOS 客户端出现 ERR_CONN_RESET_1007 时如何处理？

A Controllable Rewrite Prompt

你是企业知识库的查询改写器。

任务：结合对话历史，将用户最后一个问题改写为一条独立、可检索的查询。

规则：
1. 保留产品、平台、版本、错误码、时间等已知实体。
2. 只补全对话中已经出现的信息，不猜测缺失条件。
3. 不回答问题，不提供解决方案。
4. 如果原问题已经完整，原样返回。
5. 只输出改写后的查询。

Implementation tip: keep only the most recent relevant messages and store confirmed entities separately.

def build_rewrite_input(history, current_query, known_entities):
    return {
        "recent_history": history[-6:],
        "current_query": current_query,
        "known_entities": known_entities,
    }

Risk: Adding Unverified Conditions

Original question: “企业版怎么导出”。 A careless model might rewrite it to “企业版 Windows 客户端如何导出 PDF”, injecting “Windows” and “PDF” that were never mentioned, leading retrieval astray.

Keep both original_query and rewritten_query.

Extract entities from both and verify that key fields (product, version, date, amount) are unchanged.

For high‑risk scenarios allow only disambiguation and completion, not free expansion.

Multi‑Query: Diversify Expression

When a single rewritten query cannot cover all facets, generate several semantically related queries that approach the problem from different angles.

User asks: “Pod 一直自己重启，怎么查？”

CrashLoopBackOff 排查

容器进程异常退出

存活探针失败

OOMKilled 与进程异常退出排查

Generated multi‑query list (each query is run independently):

1. Kubernetes Pod 频繁重启的排查步骤
2. CrashLoopBackOff 常见原因及处理方法
3. 容器因健康检查失败反复重启如何定位
4. Pod OOMKilled 与进程异常退出如何排查

Results are merged, deduplicated, fused with Reciprocal Rank Fusion (RRF), and finally reranked.

def retrieve_multi_query(queries, retriever, per_query_k=10):
    result_lists = []
    for query in queries:
        docs = retriever.invoke(query)[:per_query_k]
        result_lists.append(docs)
    return reciprocal_rank_fusion(result_lists)

When to Use Multi‑Query

Terminology is inconsistent; the same concept has multiple aliases.

The problem is complex and may have several root causes.

Multi‑hop questions require evidence from different angles.

Single‑query recall remains low over time.

When Not to Use Multi‑Query

Exact look‑ups such as error codes, order numbers, API names.

The original question is already precise.

Latency‑sensitive scenarios.

Small knowledge bases where a single path already yields stable hits.

Generating 5 queries increases retrieval cost and latency; enable only after measurement shows benefit.

HyDE: Use a Hypothetical Answer as Retrieval Bridge

HyDE (Hypothetical Document Embeddings) works in three steps:

Prompt the LLM to write a plausible answer to the question.

Embed that hypothetical answer (do not treat it as factual).

Use the embedding to retrieve real documents whose language matches the answer.

问题：升级后历史数据看不到了怎么办？
↓
假设文档：升级后若历史数据暂不可见，应检查数据迁移任务、索引重建状态和租户映射，并确认旧版本数据是否完成同步……
↓ Embedding
检索真实的升级迁移与索引重建文档

HyDE is useful when the user query is short and the target documents are long, such as conceptual explanations or troubleshooting guides. It should never be used as the final answer because the generated text may contain hallucinations.

HyDE Prompt

def hyde_retrieve(question, llm, vector_store, k=20):
    hypothetical_doc = llm.invoke(
        "请为下面的问题生成一段可能出现在技术手册中的说明。"
        "内容仅用于检索，不要求事实正确，不要添加具体版本和数值。
"
        f"问题：{question}"
    ).content
    return vector_store.similarity_search(hypothetical_doc, k=k)

For precise look‑ups (e.g., contract numbers, policy dates) traditional BM25 or metadata filters are more reliable.

Choosing Between the Three Methods

问题是否依赖对话或表达不完整？
  是 → Query Rewrite
  否 →
问题是否存在多种叫法或多个检索角度？
  是 → Multi‑Query
  否 →
问题很短，且与长文档表达差距明显？
  是 → HyDE
  否 → 原查询直接检索

Typical resource profile for each strategy (LLM calls, retrieval count, main benefit, main risk):

Original query : 0 LLM calls, 1 retrieval, fast, risk – poor recall for colloquial queries.

Rewrite : 1 LLM call, 1‑2 retrievals, benefit – intent completion, risk – possible change of original meaning.

Multi‑Query : 1 LLM call, 3‑5 retrievals, benefit – broader coverage, risk – increased latency and noise.

HyDE : 1 LLM call, 1‑2 retrievals, benefit – bridges language gap, risk – hallucinated content may bias retrieval.

In practice keep the original query as a fallback path and fuse its results with those from the optimized paths.

Evaluating Query Optimization

Use a fixed evaluation set and record additional fields such as original and rewritten queries, generated queries, strategy, retrieved chunk IDs, latency, and intent‑preservation metrics.

{
  "original_query": "那苹果电脑呢",
  "rewritten_query": "macOS 客户端出现 1007 错误时如何处理",
  "generated_queries": ["...", "..."],
  "strategy": "rewrite+multi_query",
  "retrieved_chunk_ids": ["doc-42#3", "doc-19#7"],
  "latency_ms": 486
}

Key evaluation points include:

Recall@K improvement.

Rank of the correct document.

Fidelity of extracted entities (intent preservation).

Result overlap across queries.

Final answer quality.

P95 latency and cost increase.

Common Pitfalls in Production

Enabling all three methods by default inflates latency and failure surface.

Allowing the model to hallucinate missing information; missing key conditions should be clarified with the user.

Generating synonymous Multi‑Query sentences that do not add new coverage.

Treating HyDE‑generated text as factual evidence.

Comparing only final answers without analyzing contributions of each query path.

Conclusion

Query optimization bridges the gap between how users ask and how knowledge‑base documents are written. Query Rewrite fills missing context, Multi‑Query expands expression coverage, and HyDE narrows the language gap by using a document‑style hypothetical answer as a retrieval bridge.