Why Your RAG Performance Is Poor: Common Issues and Optimization Strategies

RAG Effectiveness: Common Issues and Optimization Strategies

Retrieval‑Augmented Generation (RAG) has become the dominant architecture for enterprise AI applications in 2024‑2026, with many open‑source frameworks such as LangChain RAG, LlamaIndex, RAGFlow, and QAnything. Engineers frequently spend extensive time tuning but still encounter low retrieval quality, hallucinations, and context windows filled with irrelevant content.

1. RAG Core Pipeline and 2026 Technical Evolution

A standard RAG pipeline consists of the following stages:

文档摄入 → 分块(Chunking) → 向量化(Embedding) → 存入向量数据库
                                 ↓
用户查询 → 查询向量化 → 检索 → 重排序(Rerank) → 上下文组装 → LLM生成

Key evolutions expected by 2026 include:

Embedding model upgrades : Shift from OpenAI text-embedding-ada-002 to high‑performance open‑source models such as BGE‑M3 (FlagEmbedding) and NV‑Embed‑QA, which markedly improve Chinese semantic understanding.

Hybrid retrieval as default : Combine sparse BM25 with dense vector retrieval to leverage the strengths of both.

Reranking becomes mandatory : Cross‑Encoder‑based rerankers become the industrial‑grade default component in 2025.

Graph index integration : GraphRAG shows clear advantages in multi‑hop QA and relational reasoning.

2. Root‑Cause Analysis of Retrieval Failures

2.1 Embedding Model Selection

The embedding model is the foundation of retrieval quality. Common mistakes include using OpenAI ada‑002 for Chinese documents, which lags behind dedicated Chinese models. Experiments show that text-embedding-3-small yields poor similarity distribution for technical Chinese texts.

Recommended models (dimension, Chinese capability, suitable scenarios):

BGE‑M3 (FlagEmbedding) – 1024/1536/1792 dimensions – excellent – general purpose.

NV‑Embed‑QA – 1024 dimensions – excellent – NVIDIA ecosystem.

Jina Embeddings v3 – 1024 dimensions – good – rapid prototyping.

BGE‑Large‑ZH – 1024 dimensions – excellent – pure Chinese.

Example code for loading BGE‑M3 (Python):

from FlagEmbedding import BGEM3FlagModel

# Load model with fp16 to save memory
model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)

# Encode documents
documents = ["文档内容1", "文档内容2"]
embeddings = model.encode(documents, batch_size=8)

# Encode a single query
query_embedding = model.encode_queries(["用户查询"])

Dimension choice : Higher dimensions improve semantic capacity but increase storage and latency. For vectors under one million, 1536 dimensions offer a good balance.

Embedding quality evaluation : Use the Massive Text Embedding Benchmark (MTEB); BGE‑M3 ranks in the top tier for Chinese as of April 2025.

2.2 Chunking Strategy

Chunking is the second most influential factor and is often overlooked.

Fatal flaw of fixed‑size chunking :

# Incorrect example: fixed length chunking ignoring semantic boundaries
text = document.text
chunks = [text[i:i+512] for i in range(0, len(text), 512)]

Fixed chunking can split sentences, break semantic integrity, and produce fragmented context, making it hard for the LLM to reconstruct useful information.

Semantic chunking (break on semantic distance spikes):

from langchain_experimental.text_splitter import SemanticChunker
from langchain_community.embeddings import HuggingFaceEmbeddings

splitter = SemanticChunker(
    embeddings=HuggingFaceEmbeddings(model_name="BAAI/bge-m3"),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,
)
chunks = splitter.split_text(document.text)

Hierarchical chunking for structured docs (PDF, Markdown): first split by headings, then apply semantic chunking within each level to preserve structural information.

from langchain_community.document_loaders import UnstructuredMarkdownLoader

loader = UnstructuredMarkdownLoader("文档路径", mode="elements")
documents = loader.load()

Chunk size guidelines (tokens):

Q&A, FAQ: 100‑200

Technical docs, tutorials: 300‑512

Long texts (contracts, papers): 512‑1024

3. Hybrid Retrieval Architecture

3.1 Fusion of Sparse (BM25) and Dense Retrieval

BM25 excels at exact keyword matching; dense vector retrieval captures semantic similarity. Both have blind spots.

Fusion method : Run BM25 and vector search in parallel and combine results with Reciprocal Rank Fusion (RRF):

RRF_score(d) = Σ 1/(k + rank_i(d)) for i in retrieval methods
# Default k = 60

Python example:

from rank_bm25 import BM25Okapi
import numpy as np

# BM25
tokenized_corpus = [doc.split() for doc in corpus]
bm25 = BM25Okapi(tokenized_corpus)
bm25_scores = bm25.get_scores(query.split())

# Dense (Faiss)
import faiss
dimension = 1536
index = faiss.IndexFlatIP(dimension)
index.add(np.array(embeddings).astype('float32'))
_, vector_indices = index.search(query_embedding, top_k)

# RRF fusion
def rrf_fusion(bm25_scores, vector_indices, k=60):
    rrf_scores = np.zeros(len(corpus))
    for idx, vec_idx in enumerate(vector_indices[0]):
        rrf_scores[vec_idx] += 1 / (k + idx + 1)
    sorted_bm25_indices = np.argsort(bm25_scores)[::-1]
    for idx, doc_idx in enumerate(sorted_bm25_indices[:top_k]):
        rrf_scores[doc_idx] += 1 / (k + idx + 1)
    return np.argsort(rrf_scores)[::-1]

final_indices = rrf_fusion(bm25_scores, vector_indices)

3.2 Alpha Parameter Tuning

Frameworks such as Azure AI Search and RAGFlow expose an alpha parameter to weight sparse vs. dense retrieval: alpha = 0.5 (default): equal weight. alpha → 1: favor dense retrieval (stronger semantic understanding). alpha → 0: favor BM25 (stronger exact matching).

Determine the optimal alpha by measuring Recall@K rather than guessing.

4. Reranking in Practice

Reranking is essential for industrial‑grade RAG. The initial retrieval stage aims for high recall (e.g., top‑100), while reranking refines to the top‑10 most relevant results.

4.1 Cross‑Encoder vs. Bi‑Encoder

Bi‑Encoder : Independent encoding of documents and queries; medium accuracy; fast.

Cross‑Encoder : Joint encoding; high accuracy; slower because each query‑document pair is evaluated.

4.2 Reranker Model Configuration

bge‑reranker‑v2‑m3 (FlagEmbedding) – high accuracy, medium speed.

Cohere‑rerank‑3.5 (Cohere) – high accuracy, fast API.

Jina‑reranker‑v2 (Jina) – medium‑high accuracy, fast.

Python example using a Cross‑Encoder reranker:

from sentence_transformers import CrossEncoder

# Load BGE reranker
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

# Rerank retrieved documents
pairs = [[query, doc] for doc in retrieved_documents]
scores = reranker.predict(pairs)
ranked_indices = np.argsort(scores)[::-1]
ranked_documents = [retrieved_documents[i] for i in ranked_indices]

Common rerank failures and checks:

Query‑document concatenation exceeds model max_length → truncation.

Domain‑specific poor performance → need domain fine‑tuning.

Candidate set too small (< 50) → limited benefit.

5. Query Optimization

5.1 Query Rewriting

Semantic gaps between user queries and document phrasing cause mismatches. Example rewrite function using OpenAI GPT‑4o:

from openai import OpenAI

client = OpenAI()

def rewrite_query(query):
    prompt = f"""Rewrite the following user query to be more retrieval‑friendly while preserving intent, adding synonyms and hypernyms.
    Example:
    Input: 怎么调教模型
    Output: 模型微调 fine‑tuning 指令微调

    Input: {query}
    Output:"""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return response.choices[0].message.content

5.2 HyDE (Hypothetical Document Embeddings)

Generate a hypothetical answer with an LLM, embed it, and use it for retrieval. Works well for open‑domain QA but can mislead in specialized domains.

def hyde_retrieve(query, top_k=10):
    # Step 1: Generate hypothetical answer
    hyde_prompt = f"针对以下问题，生成一段详细的技术回答：
{query}"
    hypothetical_doc = llm_generate(hyde_prompt)
    # Step 2: Retrieve with the hypothetical document
    hyde_embedding = model.encode_queries([hypothetical_doc])
    _, indices = vector_index.search(hyde_embedding, top_k)
    return indices

6. Context Window Waste

Even with correct retrieval, filling the LLM context window with irrelevant passages dilutes attention and harms answer quality.

Symptoms : High top‑1 accuracy but poor final answer; hallucinations appear despite relevant citations.

Solutions :

Context compression: Summarize retrieved chunks before feeding them to the LLM.

Fine‑grained context selection: Control chunk boundaries during retrieval; use overlapping chunks to preserve continuity.

Long‑context models: Deploy models with 128K+ windows (e.g., GPT‑4o 128K, Claude 3.5 200K) together with efficient attention mechanisms such as Longformer or streaming LLMs.

7. Evaluation Metrics and Benchmarks

Optimization requires measurement at three levels:

Retrieval layer : Recall@K (RAGAS tool).

Generation layer : Faithfulness (alignment with retrieved content) and Answer Relevance (question relevance), also via RAGAS.

Example RAGAS evaluation pipeline:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall, context_precision
from datasets import Dataset

eval_data = {
    "user_input": [q1, q2, q3],
    "retrieved_contexts": [[ctx1], [ctx2], [ctx3]],
    "response": [a1, a2, a3],
    "reference": [ref1, ref2, ref3]
}

dataset = Dataset.from_dict(eval_data)
result = evaluate(dataset, metrics=[context_recall, faithfulness, answer_relevancy])

Production validation: A/B test different RAG configurations and measure real‑user signals (likes/dislikes, follow‑up rate, task completion).

8. Troubleshooting Checklist and Best Practices

Checklist

Recall = 0 : Embedding model not loaded correctly → verify vector dimension match → regenerate all vectors.

Top‑K results irrelevant : Chunk size too small or too large → visualize results → adjust chunk size.

Semantic mismatch : Mixed Chinese/English → separate language data → use multilingual model.

Severe hallucinations : Context polluted by unrelated content → analyze attention weights → enable rerank + context compression.

High latency : Vector index not optimized → profile retrieval stage → enable HNSW index.

Best‑Practice Summary

Prioritize embedding model selection; BGE‑M3 or equivalent Chinese‑optimized models give the best cost‑performance.

Experiment with chunking; no universal setting—run A/B tests with different configurations early.

Adopt hybrid retrieval (BM25 + vector) as the industry standard.

Make reranking indispensable; it yields the largest boost in top‑1 accuracy.

Drive optimization with evaluation; maintain an offline RAGAS pipeline and complement it with online metrics.

Govern context; retrieved content must be filtered, compressed, and precisely assembled.

RAG performance issues are rarely isolated; they stem from the entire retrieval‑rerank‑assembly chain. Engineers should build a complete evaluation system and let data guide each optimization round rather than relying on intuition.