5 Production‑Ready RAG Architectures with LangGraph & LlamaIndex: From Naive to Agentic

Retrieval‑Augmented Generation (RAG) is no longer a single pipeline; production systems need to pick the right retrieval architecture for the query type. Choosing the wrong architecture can hurt latency, relevance, and even cause the system to return confidently wrong answers that amplify with scale.

Naive RAG

Naive RAG is the simplest layer: documents are loaded, a VectorStoreIndex is built, and a QueryEngine wrapped with @tool is handed to a LangGraph agent. It works well for internal policy bots, FAQ assistants, and document search where the answer resides in a single text chunk.

The pipeline has five steps—four index‑time steps (run once) and one query‑time step (run per user request). Embedding cost is paid during indexing; at query time only a similarity search and an LLM call are performed.

# Naive RAG – full template
import os
from typing import Literal, Annotated
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage, BaseMessage
from langchain_core.tools import tool
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode
from langgraph.checkpoint.memory import MemorySaver
from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex, StorageContext, load_index_from_storage
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

llm = ChatOpenAI(model="gpt-4o", temperature=0)
Settings.llm = LlamaOpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 512
Settings.chunk_overlap = 50

# Index creation (run once)
if os.path.exists("./storage/naive"):
    storage_context = StorageContext.from_defaults(persist_dir="./storage/naive")
    index = load_index_from_storage(storage_context)
else:
    documents = SimpleDirectoryReader("./data").load_data()
    index = VectorStoreIndex.from_documents(documents, show_progress=True)
    index.storage_context.persist(persist_dir="./storage/naive")

query_engine = index.as_query_engine(similarity_top_k=3)

@tool
def search_knowledge_base(query: str) -> str:
    """Search internal company documents for policies, product specs, and procedures."""
    response = query_engine.query(query)
    return str(response)

tools = [search_knowledge_base]
llm_with_tools = llm.bind_tools(tools)
tool_node = ToolNode(tools)

class State(MessagesState):
    pass

def agent_node(state: State) -> dict:
    system_prompt = SystemMessage(content="You are a helpful assistant with access to an internal knowledge base. Use search_knowledge_base for company‑specific questions.")
    response = llm_with_tools.invoke([system_prompt] + state["messages"])
    return {"messages": [response]}

def should_continue(state: State) -> Literal["tools", "__end__"]:
    last = state["messages"][-1]
    if hasattr(last, "tool_calls") and last.tool_calls:
        return "tools"
    return "__end__"

builder = StateGraph(State)
builder.add_node("agent", agent_node)
builder.add_node("tools", tool_node)
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", should_continue, {"tools": "tools", "__end__": END})
builder.add_edge("tools", "agent")
graph = builder.compile(checkpointer=MemorySaver())

Limitations: terminology mismatches (e.g., "SLA" vs. "Gold‑tier") and relational questions that span multiple chunks cannot be answered because Naive RAG returns isolated text fragments.

Hybrid RAG

Hybrid RAG combines dense vector search and sparse BM25 keyword matching to cover both semantic equivalence and exact term requirements. The two retrievers run in parallel, their candidate lists are merged, and a cross‑encoder reranker produces the final ranking.

# Hybrid RAG – LlamaIndex implementation
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from langchain_core.tools import tool

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

vector_retriever = VectorIndexRetriever(index=index, similarity_top_k=5)
bm25_retriever = BM25Retriever.from_defaults(index=index, similarity_top_k=5)

hybrid_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    similarity_top_k=3,
    mode="reciprocal_rerank",
    use_async=True,
)

hybrid_query_engine = RetrieverQueryEngine.from_args(retriever=hybrid_retriever)

@tool
def search_hybrid(query: str) -> str:
    """Search using both semantic similarity and keyword matching. More precise for technical terms, product codes, and exact names."""
    response = hybrid_query_engine.query(query)
    return str(response)

Hybrid RAG shines when documents contain a mix of free‑text and structured terminology (legal citations, medical codes, API names). Smaller chunk sizes (e.g., 256 tokens) improve BM25 signal.

Graph RAG

Graph RAG changes the data model: instead of flat text chunks, it extracts entities and relationships to build a property graph. Queries become path‑finding problems, allowing multi‑hop reasoning such as "If we retire authentication module X, which enterprise customers are affected?".

# Graph RAG – property‑graph index
from llama_index.core import SimpleDirectoryReader, PropertyGraphIndex
from llama_index.indices.property_graph import SimpleLLMPathExtractor, ImplicitPathExtractor

documents = SimpleDirectoryReader("./data").load_data()
index = PropertyGraphIndex.from_documents(
    documents,
    kg_extractors=[SimpleLLMPathExtractor(llm=Settings.llm), ImplicitPathExtractor()],
    show_progress=True,
)

graph_retriever = index.as_retriever(include_text=True, retriever_mode="hybrid", similarity_top_k=3)

@tool
def search_knowledge_graph(query: str) -> str:
    """Search the knowledge graph for relationship‑centric questions (dependencies, impact chains, organizational links)."""
    response = graph_retriever.query(query)
    return str(response)

Entity extraction is costly—each document chunk triggers an LLM call—so the graph index is built once and persisted. It is ideal for compliance, risk analysis, and any domain where answers depend on linked facts.

Advanced RAG

Advanced RAG adds post‑retrieval correction layers: query rewriting (or HyDE) to generate a better search query, query decomposition for multi‑step questions, and a cross‑encoder reranker to improve relevance. These steps are optional and can be stacked on top of Naive or Hybrid pipelines.

# Advanced RAG – query rewriting, decomposition, reranking
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.query_engine import RetrieverQueryEngine, SubQuestionQueryEngine, TransformQueryEngine
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.core.query_engine.query_transform.base import HyDEQueryTransform
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from langchain_core.tools import tool

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

base_retriever = index.as_retriever(similarity_top_k=8)
reranker = SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-2-v2", top_n=3)
reranked_engine = RetrieverQueryEngine.from_args(retriever=base_retriever, node_postprocessors=[reranker])

# Sub‑question engine (decomposition)
engine_tools = [
    QueryEngineTool(query_engine=reranked_engine, metadata=ToolMetadata(name="company_knowledge_base", description="Searches company documents for policies, procedures, and product information."))
]

decomposed_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=engine_tools, use_async=True)

# HyDE transform
hyde_transform = HyDEQueryTransform(include_original=True)
hyde_engine = TransformQueryEngine(query_engine=reranked_engine, query_transform=hyde_transform)

@tool
def search_with_decomposition(query: str) -> str:
    """Break complex questions into sub‑questions, retrieve each part, and synthesize the final answer."""
    response = decompose_engine.query(query)
    return str(response)

@tool
def search_with_hyde(query: str) -> str:
    """Generate a hypothetical document for the query, embed it, and use it for retrieval – useful for abstract or exploratory questions."""
    response = hyde_engine.query(query)
    return str(response)

Best practice: start with a reranker (the biggest single gain), then add query decomposition, and finally HyDE for abstract queries. Incrementally measure recall improvements.

Agentic RAG

Agentic RAG turns the pipeline into a loop. An LLM agent decides which retrieval tool to invoke, evaluates the answer quality, and can retry with a different tool or refined query. This self‑correction makes it suitable for high‑risk domains (legal, finance, medical) where confidence thresholds matter.

# Agentic RAG – full multi‑tool agent
# (imports omitted for brevity – same as previous sections)

# Define individual tools (vector, hybrid, graph, decomposed)
@tool
def search_documents(query: str) -> str:
    return str(vector_engine.query(query))

@tool
def search_exact_terms(query: str) -> str:
    return str(hybrid_engine.query(query))

@tool
def search_relationships(query: str) -> str:
    return str(graph_engine.query(query))

@tool
def search_complex_question(query: str) -> str:
    return str(decomposed_engine.query(query))

tools = [search_documents, search_exact_terms, search_relationships, search_complex_question]
llm_with_tools = llm.bind_tools(tools)
tool_node = ToolNode(tools)

class AgentState(MessagesState):
    retrieval_count: int = 0

def agent_node(state: AgentState) -> dict:
    system_prompt = SystemMessage(content="""You are a precise research assistant with access to four retrieval tools:
1. search_documents – semantic search
2. search_exact_terms – hybrid semantic + keyword search
3. search_relationships – graph traversal
4. search_complex_question – decomposed multi‑part retrieval
Think step by step and only answer when you have sufficient evidence.""")
    response = llm_with_tools.invoke([system_prompt] + state["messages"])
    return {"messages": [response], "retrieval_count": state.get("retrieval_count", 0)}

def should_continue(state: AgentState) -> Literal["tools", "__end__"]:
    last = state["messages"][-1]
    if hasattr(last, "tool_calls") and last.tool_calls:
        return "tools"
    return "__end__"

builder = StateGraph(AgentState)
builder.add_node("agent", agent_node)
builder.add_node("tools", tool_node)
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", should_continue, {"tools": "tools", "__end__": END})
builder.add_edge("tools", "agent")
graph = builder.compile(checkpointer=MemorySaver())

Agentic RAG incurs higher latency (multiple retrieval rounds can take 8‑12 seconds) but offers self‑correction, dynamic query reformulation, and the ability to request clarification. Production patterns include streaming intermediate steps to the UI or running the agent asynchronously.

How to Choose

There is no single "best" architecture. Start with Naive RAG for simple Q&A. If terminology mismatches appear, upgrade to Hybrid. When answers require linking facts across documents, switch to Graph. If recall or context quality is insufficient, add Advanced techniques (reranking, decomposition, HyDE). Finally, for high‑risk or ambiguous queries, employ Agentic RAG to let the LLM orchestrate multiple tools and perform self‑validation.

In practice, teams often layer these capabilities: build a persistent vector index, add a hybrid retriever, overlay a property‑graph index, and expose all as tools to a LangGraph agent. This stack covers the most common production AI retrieval needs.

Author: Bessie Delight Kekeli