Bridging the Semantic Gap in RAG: Solving Mismatched Queries and Vector Store Answers

1. Problem Analysis

RAG users frequently encounter a situation where the knowledge base contains the correct answer, yet the retrieved results are unrelated because the cosine similarity between the query embedding and document embeddings is extremely low. This issue is termed the semantic gap . The gap arises from a natural "language style mismatch": user queries are colloquial, vague, and intent‑rich, while knowledge‑base documents are formal, structured, and terminology‑dense. Embedding models compute distances in vector space, so large surface‑form differences can push semantically identical texts beyond the recall threshold.

To address this, three intuitive directions are proposed: modify the user side, modify the knowledge‑base side, or insert a translation layer between them. These directions cover almost all mainstream industry solutions.

2. Query‑Side Optimizations

Query Rewriting inserts an LLM call before retrieval to rewrite the user question into a more precise, disambiguated, and document‑style query. Example: "Apple is too expensive, what should I do?" becomes "What discount options or alternatives are available for iPhone products when the price is high?" This low‑cost step yields immediate improvement.

HyDE (Hypothetical Document Embeddings) lets the LLM first generate a hypothetical answer document, then uses its embedding for retrieval. Because the generated text mimics the formal style of knowledge‑base documents, the vector distance to relevant chunks is dramatically reduced. Example: user asks "My computer shows a blue screen, what to do?" LLM produces a document describing common BSOD causes and troubleshooting steps, which aligns closely with actual documentation.

Multi‑Query expands a single query into several sub‑queries covering different angles, retrieves results for each, and merges them. For instance, "Pros and cons of new energy vehicles" expands to "What are the advantages of electric cars?", "What drawbacks do new energy cars have?", and "How do fuel cars compare with electric cars?" LangChain’s MultiQueryRetriever implements this pattern.

These techniques can be used individually or combined (e.g., Query Rewriting followed by Multi‑Query).

3. Knowledge‑Base Optimizations

Knowledge‑base improvements are performed offline. Document Chunking should follow semantic boundaries (paragraphs, sections, headings) rather than fixed token counts; overlapping chunks help preserve context at boundaries.

Hierarchical Indexing creates a two‑level index: first a summary index for coarse filtering, then a full‑text index for fine‑grained retrieval. This approach works well for long documents.

Document Enrichment generates hypothetical user questions for each chunk using an LLM and stores them alongside the chunk. This is the reverse of HyDE: instead of generating a document from a query, it generates queries from a document, further narrowing the style gap.

4. Retrieval‑Side Optimizations

Hybrid Search combines vector semantic search with traditional BM25 keyword search. The two result sets are merged using Reciprocal Rank Fusion (RRF) , which adds the inverse ranks from each list to produce a final score—simple, parameter‑free, and robust.

Reranker (Cross‑Encoder) re‑scores the top‑k candidates by jointly encoding query and document, yielding higher precision at the cost of higher compute. Popular rerankers include Cohere Rerank, the bge‑reranker series, or using LLMs such as GPT‑4/Claude for reranking.

5. Embedding Model Optimizations

If the underlying embedding model lacks domain knowledge, Fine‑tuning on real query‑document pairs can significantly improve matching. An alternative is to select a stronger general‑purpose model; recent models like text-embedding-3-large, BGE, and M3E have shown superior performance in Chinese contexts.

6. Engineering “Combo‑Punch”

A production RAG system typically layers four defenses:

Query Stage : Apply Query Rewriting, optionally add HyDE or Multi‑Query.

Retrieval Stage : Use Hybrid Search (vector + BM25) with RRF, then Reranker for precise ranking.

Knowledge‑Base Construction : Perform semantic chunking, build hierarchical indexes, and apply Document Enrichment for high‑frequency scenarios.

Continuous Optimization : Analyze online bad cases, fine‑tune embeddings, and regularly refresh the knowledge base.

The first layer is cheapest and fastest for MVP; later layers require more investment but raise the performance ceiling for mature products.

Evaluation Framework is essential: maintain a labeled test set of "query‑expected document" pairs and measure Recall@K, MRR, etc., running regression tests after each iteration.