Why Your RAG Falls Short and How to Fix It: Common Pitfalls and Proven Optimizations

1. RAG Core Pipeline and 2026 Technical Evolution

A standard RAG pipeline consists of document ingestion → chunking → embedding → vector store, followed by user query → query embedding → retrieval → rerank → context assembly → LLM generation. Recent 2026 trends include upgraded embedding models (e.g., BGE‑M3, NV‑Embed‑QA), mandatory hybrid retrieval, default use of cross‑encoder rerankers, and the emergence of GraphRAG for multi‑hop reasoning.

2. Embedding Model Selection

Common mistake: using OpenAI text‑embedding‑ada‑002 for Chinese documents, which yields poor semantic similarity.

Recommended models (as of 2025):

BGE‑M3 (FlagEmbedding) – 1024/1536/1792 dimensions, excellent Chinese understanding, general‑purpose default.

NV‑Embed‑QA – 1024 dimensions, excellent, optimized for NVIDIA ecosystem.

Jina Embeddings v3 – 1024 dimensions, good for rapid prototyping.

BGE‑Large‑ZH – 1024 dimensions, excellent for pure Chinese scenarios.

Example Python code to load BGE‑M3 with fp16:

from FlagEmbedding import BGEM3FlagModel

# Load model with half‑precision to save VRAM
model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)

documents = ["Document content 1", "Document content 2"]
embeddings = model.encode(documents, batch_size=8)

query_embedding = model.encode_queries(["User query"])

Dimension choice: 1536‑dimensional vectors give the best trade‑off for collections under one million vectors; higher dimensions improve semantic capacity but increase storage and latency.

Quality evaluation: Use the Massive Text Embedding Benchmark (MTEB); BGE‑M3 ranks in the top tier for Chinese as of April 2025.

3. Chunking Strategies

Fixed‑size chunk pitfall: splitting by a constant token count (e.g., 512) can cut sentences, break semantic coherence, and produce fragmented context.

# Incorrect example: fixed‑length chunking
text = document.text
chunks = [text[i:i+512] for i in range(0, len(text), 512)]

Semantic chunking: compute sentence‑level embedding distances and insert breaks where the distance exceeds a threshold.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_community.embeddings import HuggingFaceEmbeddings

splitter = SemanticChunker(
    embeddings=HuggingFaceEmbeddings(model_name="BAAI/bge-m3"),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,
)
chunks = splitter.split_text(document.text)

Hierarchical chunking: first split by document headings, then apply semantic chunking within each section to preserve structure.

from langchain_community.document_loaders import UnstructuredMarkdownLoader

loader = UnstructuredMarkdownLoader("path/to/document.md")
documents = loader.load()

Chunk size recommendations:

Q&A / FAQ: 100–200 tokens.

Technical docs / tutorials: 300–512 tokens.

Long‑form analysis (contracts, papers): 512–1024 tokens.

4. Hybrid Retrieval Architecture

BM25 (sparse) excels at exact keyword matches, while dense vector search captures semantic similarity. Combining both mitigates each method’s blind spots.

Reciprocal Rank Fusion (RRF) formula:

RRF_score(d) = Σ 1 / (k + rank_i(d))   for i in retrieval_methods
# default k = 60

Python example that runs BM25 and FAISS dense search in parallel and fuses results:

from rank_bm25 import BM25Okapi
import numpy as np
import faiss

# BM25 sparse retrieval
tokenized_corpus = [doc.split() for doc in corpus]
bm25 = BM25Okapi(tokenized_corpus)
bm25_scores = bm25.get_scores(query.split())

# Dense FAISS retrieval
dimension = 1536
index = faiss.IndexFlatIP(dimension)
index.add(np.array(embeddings).astype('float32'))
_, vector_indices = index.search(query_embedding, top_k)

def rrf_fusion(bm25_scores, vector_indices, k=60):
    rrf_scores = np.zeros(len(corpus))
    for idx, vec_idx in enumerate(vector_indices[0]):
        rrf_scores[vec_idx] += 1 / (k + idx + 1)
    sorted_bm25 = np.argsort(bm25_scores)[::-1]
    for idx, doc_idx in enumerate(sorted_bm25[:top_k]):
        rrf_scores[doc_idx] += 1 / (k + idx + 1)
    return np.argsort(rrf_scores)[::-1]

final_indices = rrf_fusion(bm25_scores, vector_indices)

Alpha Parameter Tuning

alpha = 0.5 (default): equal weight for BM25 and dense.

alpha → 1: favor dense retrieval for stronger semantic understanding.

alpha → 0: favor BM25 for precise keyword matching.

Determine the optimal alpha by measuring Recall@K on a validation set.

5. Reranking in Practice

Reranking refines the top‑100 initial results to a high‑quality top‑10. Two encoder families are compared:

Bi‑Encoder: independent encoding of query and documents; medium accuracy, fast.

Cross‑Encoder: joint encoding; high accuracy, slower.

Popular reranker models:

bge‑reranker‑v2‑m3 (FlagEmbedding) – high accuracy, medium speed.

Cohere‑rerank‑3.5 – high accuracy, fast API response.

Jina‑reranker‑v2 – medium‑high accuracy, fast.

Example using a BGE cross‑encoder reranker:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
pairs = [[query, doc] for doc in retrieved_documents]
scores = reranker.predict(pairs)
ranked_indices = np.argsort(scores)[::-1]
ranked_documents = [retrieved_documents[i] for i in ranked_indices]

Rerank troubleshooting:

Combined query‑document length exceeds model max_length → truncation.

Domain‑specific queries perform poorly → consider domain fine‑tuning.

Candidate set too small (< 50) → limited benefit from rerank.

6. Query Rewriting

Bridging the semantic gap between user phrasing and document terminology improves recall. Example using OpenAI gpt‑4o to rewrite queries:

from openai import OpenAI

client = OpenAI()

def rewrite_query(query):
    prompt = f"Rewrite the following user query for better retrieval while preserving intent and adding synonyms.
Input: {query}
Output:"
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return response.choices[0].message.content

7. HyDE (Hypothetical Document Embeddings)

Generate a hypothetical answer with an LLM, embed that synthetic document, and use it for retrieval. This boosts open‑domain QA but can mislead in highly specialized domains (legal, medical).

def hyde_retrieve(query, top_k=10):
    hyde_prompt = f"Provide a detailed technical answer for the question:
{query}"
    hypothetical_doc = llm_generate(hyde_prompt)
    hyde_embedding = model.encode_queries([hypothetical_doc])
    _, indices = vector_index.search(hyde_embedding, top_k)
    return indices

8. Context Window Waste

Even with correct retrieval, irrelevant filler can consume the LLM’s context window, causing hallucinations and low answer quality.

Symptom: high top‑1 accuracy but poor final answers, hallucinations appear in seemingly factual passages.

Solution 1 – Context compression: summarize retrieved chunks before feeding them to the LLM.

from langchain.chains import StuffDocumentsChain
from langchain.prompts import PromptTemplate

compress_prompt = PromptTemplate.from_template(
    """Answer the user question using only the directly relevant parts of the following context.
Context: {context}
Question: {question}
Answer:"""
)

Solution 2 – Refine window boundaries: use overlapping chunk strategy to keep semantic continuity while avoiding large irrelevant spans.

Solution 3 – Long‑context models: switch to models with 128K+ windows (e.g., GPT‑4o 128K, Claude 3.5 200K) and apply efficient attention mechanisms such as Longformer or StreamingLLM when processing very long documents.

9. Evaluation Metrics and Benchmarks

Three evaluation layers are recommended:

Retrieval layer: Recall@K (measured with RAGAS).

Generation layer – Faithfulness: alignment between generated answer and retrieved context (RAGAS).

Generation layer – Answer Relevance: relevance of the answer to the original question (RAGAS).

Sample RAGAS evaluation pipeline:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

eval_data = {
    "user_input": [q1, q2, q3],
    "retrieved_contexts": [[ctx1], [ctx2], [ctx3]],
    "response": [a1, a2, a3],
    "reference": [ref1, ref2, ref3],
}

dataset = Dataset.from_dict(eval_data)
result = evaluate(dataset, metrics=[context_recall, faithfulness, answer_relevancy])

Validate configurations in production via A/B testing (likes/dislikes, follow‑up rate, task completion) to complement offline metrics.

10. Troubleshooting Checklist

Recall = 0: embedding model not loaded correctly → verify vector dimensions and regenerate vectors.

Top‑K results irrelevant: chunk size too small/large → visualize results and adjust chunk size.

Semantic mismatch: mixed Chinese/English data → separate languages or use multilingual model.

Severe hallucination: context polluted with unrelated text → inspect attention weights and enable rerank + context compression.

High latency: vector index not optimized → profile retrieval stage and enable HNSW index.

11. Best‑Practice Summary

Prioritize high‑quality Chinese embedding models (e.g., BGE‑M3) before any other tuning.

Experiment with chunking strategies; no one‑size‑fits‑all configuration.

Adopt hybrid BM25 + dense retrieval as the industry baseline.

Rerank is essential for boosting top‑1 accuracy.

Drive improvements with systematic evaluation (RAGAS) and online metrics.

Govern context: compress, refine windows, and consider long‑context models when needed.

RAG performance issues stem from the entire retrieval‑rerank‑assembly chain. By methodically selecting embeddings, tuning chunking, employing hybrid search, applying robust reranking, and continuously evaluating, engineers can transform a flaky RAG system into a reliable AI assistant.