Boost RAG Accuracy from 60% to 94% with 11 Proven Strategies

Fundamental Problems of Naïve RAG

Naïve RAG splits documents into fixed‑size chunks, embeds them, retrieves the nearest vectors, and feeds the concatenated chunks to a large language model. This pipeline typically yields ~60% accuracy because:

Fixed‑size chunking cuts sentences in half, losing context.

Single query perspective misses alternative phrasings in the corpus.

No relevance filtering returns the mathematically nearest vectors, not the most semantically relevant.

Limited context small chunks lack a complete picture.

These failures turn a RAG system into a high‑stakes guessing game.

Eleven Effective Strategies

Strategy 1 – Context‑Aware Chunking

Split documents at semantic boundaries (headings, paragraphs) instead of fixed character counts. This preserves context such as "CEO announced…[cut]…revenue grew 40%".

from docling.chunking import HybridChunker
from transformers import AutoTokenizer

class SmartChunker:
    def __init__(self, max_tokens=512):
        self.tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
        self.chunker = HybridChunker(tokenizer=self.tokenizer, max_tokens=max_tokens, merge_peers=True)
    def chunk_document(self, document):
        chunks = list(self.chunker.chunk(dl_doc=document))
        contextualized_chunks = []
        for chunk in chunks:
            contextualized_chunks.append(self.chunker.contextualize(chunk=chunk))
        return contextualized_chunks

Pros : free, fast, preserves document structure, works with any embedding model.

Cons : slightly slower than fixed chunking, requires proper document parsing.

Strategy 2 – Contextual Retrieval

Add a short LLM‑generated summary to each chunk before embedding, making each chunk self‑contained.

async def enrich_chunk(chunk: str, document: str, title: str) -> str:
    """Add a contextual prefix using an LLM"""
    prompt = f"Title: {title}
{document[:4000]}
{chunk}

Provide a brief 1‑2 sentence context explaining this chunk."
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=150,
    )
    context = response.choices[0].message.content.strip()
    return f"{context}

{chunk}"

Pros : reduces retrieval failures by 35‑49% (Anthropic), makes chunks self‑contained, works for vector and keyword search.

Cons : extra LLM call per chunk, higher latency and cost, larger index size.

Strategy 3 – Re‑ranking

Two‑stage retrieval: fast vector search returns a larger candidate set, then a cross‑encoder re‑ranks the candidates for higher precision.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

async def search_with_reranking(query: str, limit: int = 5) -> list:
    candidate_limit = min(limit * 4, 20)
    query_embedding = await embedder.embed_query(query)
    candidates = await db.query(
        "SELECT content, metadata FROM chunks ORDER BY embedding $1 LIMIT $2",
        query_embedding, candidate_limit)
    pairs = [[query, row['content']] for row in candidates]
    scores = reranker.predict(pairs)
    reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:limit]
    return [doc for doc, score in reranked]

Pros : significantly improves precision without overloading the LLM.

Cons : slower than pure vector search, requires extra compute and modest cost.

Strategy 4 – Query Expansion

Use an LLM to transform a short user query into a detailed, context‑rich version.

async def expand_query(query: str) -> str:
    """Expand a short query into a detailed version"""
    system_prompt = "You are a query‑expansion assistant. Expand the query with context, relevant terms, and clarification."
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": system_prompt}, {"role": "user", "content": f"Expand this query: {query}"}],
        temperature=0.3,
    )
    return response.choices[0].message.content.strip()

Example :

Input: "What is RAG?"
Output: "What is Retrieval‑Augmented Generation (RAG), how does it combine information retrieval with language generation, what are its key components and advantages for QA systems?"

Pros : improves recall for vague queries, fast. Cons : extra LLM call adds latency, may over‑specify simple queries.

Strategy 5 – Multi‑Query RAG

Generate 3‑4 paraphrases of the original query, search all in parallel, and deduplicate results.

async def search_with_multi_query(query: str, limit: int = 5) -> list:
    variations_prompt = f"Generate 3 different formulations for the query:

\"{query}\"

Return only the queries, one per line."
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": variations_prompt}],
        temperature=0.7,
    )
    queries = [query] + response.choices[0].message.content.strip().split('
')
    # Parallel search omitted for brevity
    return results

Pros : better recall for ambiguous queries, captures multiple perspectives. Cons : up to 4× database queries, higher API cost, possible duplicate results.

Strategy 6 – Agent‑Based RAG

Provide an AI agent with multiple retrieval tools (semantic search, full‑document fetch, SQL query) and let it choose the appropriate tool per query.

from pydantic_ai import Agent

agent = Agent('openai:gpt-4o', system_prompt='You are a RAG assistant with multiple retrieval tools.')

@agent.tool
async def search_knowledge_base(query: str, limit: int = 5) -> str:
    query_embedding = await embedder.embed_query(query)
    results = await db.match_chunks(query_embedding, limit)
    return format_results(results)

@agent.tool
async def retrieve_full_document(document_title: str) -> str:
    result = await db.query("SELECT title, content FROM documents WHERE title ILIKE %s", f"%{document_title}%")
    return f"**{result['title']}**

{result['content']}"

Pros : highly flexible, can handle heterogeneous data sources. Cons : more complex to implement, less predictable behavior, higher latency.

Strategy 7 – Self‑Reflective RAG

After an initial retrieval, grade relevance with an LLM. If the grade is low, refine the query and search again (up to a configurable number of iterations).

async def search_with_self_reflection(query: str, limit: int = 5, max_iterations: int = 2) -> dict:
    for iteration in range(max_iterations):
        results = await vector_search(query, limit)
        grade_prompt = f"Query: {query}

Retrieved docs:
{format_docs_for_grading(results)}
Rate relevance 1‑5. Respond with a number only."
        grade_resp = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": grade_prompt}],
            temperature=0,
        )
        grade = int(grade_resp.choices[0].message.content.strip())
        if grade >= 3:
            return {"results": results, "iterations": iteration + 1, "final_query": query}
        if iteration < max_iterations - 1:
            refine_prompt = f"The query \"{query}\" returned low relevance. Propose an improved query."
            refined = await client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": refine_prompt}],
                temperature=0.5,
            )
            query = refined.choices[0].message.content.strip()
    return {"results": results, "iterations": max_iterations, "final_query": query}

Pros : self‑correction, can recover from poor initial results. Cons : highest latency (multiple LLM calls), most expensive, may still fail on very hard queries.

Strategy 8 – Knowledge‑Graph RAG

Combine vector search with a graph database (Neo4j) to capture entity relationships that pure text similarity misses.

from graphiti_core import Graphiti
from graphiti_core.nodes import EpisodeType

graphiti = Graphiti("neo4j://localhost:7687", "neo4j", "password")

async def ingest_document(text: str, source: str):
    await graphiti.add_episode(name=source, episode_body=text, source=EpisodeType.text)

async def search_knowledge_graph(query: str) -> str:
    results = await graphiti.search(query=query, num_results=5)
    formatted = []
    for r in results:
        formatted.append(f"Entity: {r.node.name}
Type: {r.node.type}
Relations: {r.relationships}")
    return "
---
".join(formatted)

Pros : captures relationships missed by pure vector search, reduces hallucinations, ideal for interconnected data. Cons : requires Neo4j infrastructure, more complex setup, slower and costlier, needs entity extraction.

Strategy 9 – Hierarchical RAG

Create parent‑child chunk relationships. Search fine‑grained child chunks for relevance, then return the larger parent chunk for context.

def ingest_hierarchical(document: str, title: str):
    parent_chunks = [document[i:i+2000] for i in range(0, len(document), 2000)]
    for pid, parent in enumerate(parent_chunks):
        metadata = {"heading": f"{title} - Part {pid}"}
        db.execute("INSERT INTO parent_chunks (id, content, metadata) VALUES (%s, %s, %s)", (pid, parent, json.dumps(metadata)))
        child_chunks = [parent[j:j+500] for j in range(0, len(parent), 500)]
        for child in child_chunks:
            embedding = get_embedding(child)
            db.execute("INSERT INTO child_chunks (content, embedding, parent_id) VALUES (%s, %s, %s)", (child, embedding, pid))

async def hierarchical_search(query: str) -> str:
    query_emb = get_embedding(query)
    results = await db.query("""
        SELECT p.content, p.metadata
        FROM child_chunks c
        JOIN parent_chunks p ON c.parent_id = p.id
        ORDER BY c.embedding %s LIMIT 3
    """, query_emb)
    formatted = []
    for content, metadata in results:
        meta = json.loads(metadata)
        formatted.append(f"[{meta['heading']}]
{content}")
    return "

".join(formatted)

Pros : balances precision with context, reduces noise, natural for structured documents. Cons : requires parent‑child indexing, more complex ingestion, careful hierarchy design.

Strategy 10 – Late Chunking

Embed the whole document first, then slice it into chunks while preserving the full‑document context in each chunk embedding.

def late_chunk(text: str, chunk_size=512):
    """Chunk after full‑document embedding"""
    full_doc_embeddings = transformer_embed(text)  # token‑level embeddings
    tokens = tokenize(text)
    chunks = []
    for start in range(0, len(tokens), chunk_size):
        end = start + chunk_size
        chunk_text = detokenize(tokens[start:end])
        chunk_emb = mean_pool(full_doc_embeddings[start:end])
        chunks.append((chunk_text, chunk_emb))
    return chunks

Pros : retains complete document context, leverages long‑context models, better semantic understanding. Cons : needs a long‑context embedding model, more complex implementation, limited by model max token length.

Strategy 11 – Fine‑Tuned Embeddings

Train an embedding model on domain‑specific query‑document pairs to improve understanding of specialized terminology.

from sentence_transformers import SentenceTransformer, losses
from torch.utils.data import DataLoader

def prepare_training_data():
    return [
        ("What is EBITDA?", "EBITDA (Earnings Before Interest, Taxes, Depreciation, and Amortization) ..."),
        ("Explain capital expenditures", "Capital expenditures (CapEx) refer to ..."),
        # ... thousands more pairs
    ]

def fine_tune_model():
    model = SentenceTransformer('all-MiniLM-L6-v2')
    train_examples = prepare_training_data()
    train_loader = DataLoader(train_examples, shuffle=True, batch_size=16)
    train_loss = losses.MultipleNegativesRankingLoss(model)
    model.fit(train_objectives=[(train_loader, train_loss)], epochs=3, warmup_steps=100)
    model.save('./fine_tuned_financial_model')
    return model

embedding_model = SentenceTransformer('./fine_tuned_financial_model')

Pros : typically improves accuracy by 5‑10%, understands domain terminology, smaller models can outperform larger generic ones. Cons : requires labeled data, training time and resources, periodic re‑training.

Combining Strategies for Maximum Impact

Individual strategies help, but their combination is transformative.

Combo 1 – Production‑Ready Stack (Best Overall)

Context‑Aware Chunking

Re‑ranking

Query Expansion

Agent‑Based RAG

Result: ~92% accuracy, ~1.2 s latency, ~$0.003 per query.

Combo 2 – High‑Accuracy Stack (Critical Applications)

Contextual Retrieval

Multi‑Query RAG

Re‑ranking

Self‑Reflective RAG

Result: ~96% accuracy, ~2.5 s latency, ~$0.008 per query – suitable for medical, legal, or financial use cases.

Combo 3 – Domain‑Expert Stack (Specialized Fields)

Fine‑Tuned Embeddings

Contextual Retrieval

Knowledge Graph

Re‑ranking

Result: ~94% accuracy on domain queries, ~1.8 s latency, ~$0.005 per query after initial training investment.

Implementation Roadmap

Stage 1 – Foundation (Week 1) : Switch to context‑aware chunking, set up basic vector search, benchmark baseline accuracy.

Stage 2 – Quick Wins (Weeks 2‑3) : Add re‑ranking and query expansion, measure improvements.

Stage 3 – Advanced (Weeks 4‑6) : Introduce multi‑query or agent‑based RAG, implement self‑reflection for critical queries, start fine‑tuning embeddings.

Stage 4 – Specialization (Month 2+) : Add contextual retrieval for high‑value docs, consider knowledge‑graph integration, fine‑tune embeddings for the target domain.

Real‑World Results

E‑commerce chatbot : Accuracy rose from 58% to 91%, support‑ticket escalation dropped from 35% to 12% (Combo 1).

Medical document system : Accuracy improved from 62% to 96%, approved for clinical use (Combo 2).

Legal contract analysis : Accuracy increased from 65% to 94%, contract review time cut by 60% (Combo 3).

Common Pitfalls to Avoid

Implementing all strategies at once – leads to complexity and high cost.

Skipping baseline measurements – you cannot prove improvement without metrics.

Using fixed‑size chunks – destroys semantic continuity.

Omitting re‑ranking – vector similarity alone often yields sub‑optimal relevance.

Neglecting query preprocessing – vague queries cause retrieval failures.

Relying on a single retrieval method – diverse queries benefit from flexible, multi‑tool agents.

Future Trends in RAG

Smaller, faster models : new embedding models promise 10× speed with comparable accuracy.

Multimodal RAG : retrieval of images, tables, and charts alongside text.

Sparse‑plus‑dense retrieval : techniques like SPLADE combine neural and sparse signals.

Conclusion

Building a production‑ready RAG system is less about chasing the flashiest technology and more about systematically fixing the failure modes of naïve RAG. Starting with context‑aware chunking and re‑ranking, then layering query expansion, multi‑query, self‑reflection, or domain‑specific enhancements, yields dramatic accuracy gains—from 60% to over 94% in the experiments presented. Measure every change, iterate based on real performance data, and only add complexity when the benefits outweigh the costs.

References

GitHub repository – Advanced RAG Strategies: https://github.com/coleam00/ottomator-agents/tree/main/all-rag-strategies Anthropic Contextual Retrieval research: https://www.anthropic.com/news/contextual-retrieval Pinecone Re‑ranking guide: https://www.pinecone.io/learn/series/rag/rerankers/ Docling context‑aware chunking: https://github.com/DS4SD/docling Graphiti knowledge‑graph RAG: https://github.com/graphiti-ai/graphiti