Transform a Single RAG Pipeline with LangGraph – Agent Picks Vector, Graph or Web Search

Traditional RAG pipelines follow a fixed sequence: receive a question, retrieve documents, and produce an answer, regardless of the query type. This design fails when different questions require different retrieval strategies such as semantic vector search, graph relationship queries, or real‑time web lookup.

LangGraph enables an Agentic RAG architecture by modeling the agent as a directed graph (state machine). Nodes represent functional steps and edges can form loops, allowing the system to remember previous actions, make decisions, and retry until a confident answer is generated.

Key Components

Router : the entry node that classifies the incoming query into one of four categories— vector, graph, web, or direct —using a structured‑output LLM call ( with_structured_output). The classification becomes a conditional edge that determines the next retriever.

Retriever nodes :

Vector retriever (FAISS/pgvector) for factual questions.

Graph retriever (Neo4j + GraphCypherQAChain) for relationship queries.

Web retriever (Tavily API) for up‑to‑date information.

Direct node that calls the LLM directly for simple calculations or known facts.

Grader : evaluates each retrieved document with a structured LLM output ( relevant or irrelevant). If at least one document is relevant, the flow proceeds to generation; otherwise it may trigger rewriting or fall back to web search.

Rewriter : when no relevant documents are found, the LLM rewrites the query to be more specific, increments a rewrite_count (max 3), and routes the new query back to the Router.

Generator : concatenates the content of all documents marked relevant and asks the LLM to answer using only that context.

Hallucination Checker : a final verification step that asks a structured LLM whether every statement in the generated answer is supported by the retrieved context. If not, the system regenerates the answer with a stricter prompt.

State Definition

from typing import TypedDict, List, Literal
from langchain_core.documents import Document
from langgraph.graph import StateGraph, END

class AgentState(TypedDict):
    question: str
    route: str            # vector | graph | web | direct
    documents: List[Document]
    generation: str
    rewrite_count: int
    grade_results: List[str]

The workflow is assembled by adding each node to a StateGraph, setting the Router as the entry point, and defining conditional edges based on the Router’s decision, the Grader’s outcome, and the Hallucination Checker’s result.

# Build the graph
workflow = StateGraph(AgentState)
workflow.add_node("router", router_node)
workflow.add_node("vector", vector_node)
workflow.add_node("graph", graph_node)
workflow.add_node("web", web_node)
workflow.add_node("direct", direct_node)
workflow.add_node("grader", grader_node)
workflow.add_node("rewriter", rewriter_node)
workflow.add_node("generator", generator_node)
workflow.add_node("hallucination", hallucination_node)
workflow.set_entry_point("router")
# Conditional edges omitted for brevity
agent = workflow.compile()
result = agent.invoke({"question": "What is our data retention policy for EU customers?", "rewrite_count": 0, "documents": [], "grade_results": [], "generation": "", "route": ""})
print(result["generation"])

Running the agent on example queries such as “What is our refund policy?” (vector), “Which top‑3 customers are linked to which suppliers?” (graph), or “What did the Fed announce this morning?” (web) shows that the Router correctly selects the appropriate retriever, the Grader filters irrelevant results, the Rewriter refines failed queries, and the Hallucination Checker prevents unsupported answers.

In summary, by converting a single‑pass RAG pipeline into a LangGraph‑driven state machine, the system gains memory, decision‑making, and self‑correction capabilities, effectively breaking the limitations of traditional pipelines and delivering more reliable, context‑grounded answers.