How Query Rewriting Boosts Retrieval in RAG Systems

Why Query Rewriting Is Needed

In real‑world RAG deployments, user queries frequently suffer from unclear intent, which limits the benefit of even the best retrieval techniques. Rewriting the query before retrieval increases the chance of retrieving the most relevant content blocks.

Typical Query Defects

Vague expression : colloquial or omitted key entities.

Unclear references : pronouns in multi‑turn dialogue cannot be used directly for retrieval.

Vocabulary gap : everyday language versus domain‑specific indexing terms.

Implicit intent : the real information need is not explicitly stated.

RAG is often used in multi‑turn conversations where such issues make it hard for the system to understand user intent.

Retriever Defects

Both dense vector retrieval and sparse keyword retrieval share a fundamental limitation: the retriever only passively computes similarity between query and documents and lacks proactive understanding or completion of query intent, making retrieval quality highly dependent on query quality.

Sparse retrieval (e.g., BM25) is limited to exact term matching and struggles with synonyms or polysemy; dense retrieval captures semantics but remains sensitive to wording, negation, and detail constraints.

Consequently, even a perfect index and ranking algorithm cannot compensate for low‑quality original queries.

Traditional Query Rewriting Solutions

Query Normalization

Basic rule‑based rewriting applicable to most production systems.

Full‑width to half‑width conversion: "ＡＩ技术" → "AI技术"

Traditional‑simplified Chinese conversion: "機器學習" → "机器学习"

Case normalization: "GOOD job" → "good job"

Stop‑word removal: delete words like "的", "了", "请问", "帮我"

Punctuation normalization: remove extra exclamation marks, ellipses, etc.

Spelling correction

Edit‑distance correction: "机器鞋习" → "机器学习"

Language‑model‑based correction: context‑aware, handles homophones such as "人工指能" → "人工智能"

Keyboard‑distance model: corrects errors from pinyin input methods

Synonym and Dictionary Expansion

Synonym dictionary mapping: "购买" → "买入/采购/入手/下单"

Domain‑specific term mapping: "心梗" → "心肌梗死"; acronym expansion "AI" → "人工智能"

Hypernym/hyponym expansion: generalize "德牧" → "大型犬" or specialize "手机" → "iPhone/华为/小米" to adjust recall granularity

Frequency‑Based Rewriting

TF‑IDF keyword extraction: extract high‑weight terms to reconstruct short queries

Query Likelihood + PRF (e.g., RM3): retrieve top‑k documents, extract frequent terms, and expand the query

Pointwise Mutual Information (PMI) expansion: add terms with highest PMI to query terms, e.g., "苹果 手机" → "iOS/屏幕/摄像头"

Linguistic Rule Rewriting

Lemma and stem extraction: "购买了/购买过/买了" → "购买"

Syntactic transformation: convert question to statement, e.g., "如何治疗高血压？" → "高血压治疗方法" (handle negation carefully)

Entity recognition + rule replacement: map "去年" to a specific year, fill location for "这里", expand "几百万" to a range query

Template‑Based Rewriting

Intent template matching: replace with predefined patterns, e.g., "{产品名} 使用教程/使用方法/操作指南"

Slot filling: after recognizing intent and slots, generate "北京 明天 天气预报" from

query_weather + location=北京, time=明天

Modern Query Rewriting Solutions

LLM inference capabilities have become strong, making LLM‑based rewriting mature.

Generation‑Based Single‑Query Rewriting

HyDE (Hypothetical Document Embeddings) : the LLM first generates a hypothetical answer document, then uses its embedding for retrieval, shifting the query space toward the document side and significantly improving semantic alignment.

Reasoning‑Enhanced Rewriting : instead of directly outputting a new query, the model performs chain‑of‑thought reasoning inside a <think> tag before emitting the final rewritten query, which is especially useful for multi‑hop inference problems.

Query Decomposition

Complex, multi‑hop questions often cannot be satisfied by a single retrieval. Decomposition splits a complex question into independent sub‑questions, each processed by RAG, and then merges the results.

Example: Original query “阿良是谁，他后来恢复十四境了没有？” becomes sub‑question 1 “阿良是谁？” and sub‑question 2 “阿良后来恢复十四境了没有？”

Note: Decomposition differs from parallel execution of independent simple queries such as “阿良是谁？陈平安是谁？” which should be classified as multi‑query parallelism, not decomposition.

Multi‑View Rewriting

Generate multiple query variants from different perspectives and fuse the results to boost recall.

Dynamic Rewriting in Agentic RAG

In an Agentic RAG workflow, query rewriting can shift from a one‑off static operation to a dynamic step within an autonomous reasoning loop.

Because Agentic RAG continuously adapts its retrieval strategy, if the initial results are incomplete or irrelevant, the agent rewrites the query, adjusts the retrieval plan, or performs multi‑hop retrieval until it reaches sufficient confidence or hits a budget limit.

Compared with a fixed “retrieve‑then‑generate” pipeline, Agentic RAG integrates planning, retrieval, reasoning, critique, rewriting, and reflection in a self‑contained loop, where rewriting may be exposed as an independent tool or performed internally on the query string.

Conclusion

Don’t Rewrite Blindly

Over‑expansion: adding too many terms introduces noise, reduces precision, and raises cost.

Semantic drift: LLM rewriting may alter the original meaning or hallucinate.

Latency and cost overhead: extra LLM inference increases response time and expense.

Query rewriting is not guaranteed to improve retrieval; it can sometimes backfire.

Strategy Selection

User queries are raw material; they may need normalization, expansion, or even rejection, prompting the user to restate a clearer question.

LLMs can transform vague problems into structured queries that contain explicit entities and constraints.

Evaluation Framework

Retrieval‑layer metrics: Recall@K, MRR, NDCG

Generation‑layer metrics: monitor retrieval failures, model ignoring retrieval context, contradictory document information, and knowledge‑base gaps.

Building an automated query‑rewriting evaluation pipeline is essential for systematically discovering issues.