Beyond Vector Retrieval: Building a Multi‑Strategy RAG Agent with LangGraph

Problem with Traditional Pipeline RAG

Conventional RAG systems—semantic search, vector DB migration, Graph RAG—operate as fixed pipelines that process any query in the same sequence, regardless of the query type. Real‑world questions often require different tools: a policy question needs vector search, a relationship query needs graph traversal, and a news‑related question needs web search.

"What is our refund policy?" → Vector search "What is the relationship between top customers and this supplier?" → Graph traversal "What did the Federal Reserve announce this morning?" → Web search

Pipeline architectures can handle only one of these; an agent can handle all and decide which tool to use.

What Makes an Agent "Agentic"?

The agentic RAG model is a state machine with memory, decision‑making, and self‑correction, terminating only when the answer is confident. Four capabilities differentiate it from a pipeline: dynamic routing, relevance grading, query rewriting, and hallucination checking.

Routing Demonstration

Examples of the router selecting the appropriate tool:

"What is our data retention policy?" → Vector

"Which top‑3 customers are linked to which suppliers?" → Graph

"What is the latest Fed announcement on rates?" → Web

"What is 15% of 340?" → Direct LLM

LangGraph State Machine

LangGraph models the agent as a directed graph where nodes are functions and edges are decisions. Unlike LangChain, edges can form cycles, allowing the agent to retry, rewrite, and self‑correct until a confident answer is produced.

Node Descriptions

Router : The decision hub that classifies the query into one of four categories (vector, graph, web, direct) using a structured LLM output. The classification becomes a conditional edge that determines the next node.

Retriever Nodes :

Vector: FAISS/pgvector cosine similarity search.

Graph: Neo4j with auto‑generated Cypher queries.

Web: Tavily (or SerpAPI) for real‑time results.

Direct: Calls the LLM directly for calculations or general knowledge.

Grader : Evaluates each retrieved document for relevance. If at least one document is relevant, the flow proceeds to the Generator; otherwise, it may trigger the Rewriter or fall back to Web after three failed attempts.

Rewriter : When retrieval fails, the LLM rewrites the query to be more specific, increments a rewrite counter (max 3), and routes back to the Router.

Generator : Generates an answer using only documents marked relevant, with a strict prompt: "Answer using only the context below." The answer and source documents are stored in the agent state for later verification.

Hallucination Checker : Verifies that every claim in the generated answer is supported by the retrieved context. If not, it forces a regeneration with a stricter prompt.

Code Implementation – Step‑by‑Step

Step 1 – Define Agent State

from typing import TypedDict, List, Literal
from langchain_core.documents import Document
from langgraph.graph import StateGraph, END
from langchain_openai import ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
from langchain_community.tools.tavily_search import TavilySearchResults
from langchain_community.graphs import Neo4jGraph
from pydantic import BaseModel

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

class AgentState(TypedDict):
    question: str
    route: str  # vector | graph | web | direct
    documents: List[Document]
    generation: str
    rewrite_count: int
    grade_results: List[str]

embeddings = OpenAIEmbeddings()
vector_store = FAISS.load_local("faiss_index", embeddings)
vector_retriever = vector_store.as_retriever(search_kwargs={"k": 4})
neo4j_graph = Neo4jGraph(url="bolt://localhost:7687", username="neo4j", password="password")
web_search = TavilySearchResults(max_results=3)

Step 2 – Router Node

class RouteDecision(BaseModel):
    route: Literal["vector", "graph", "web", "direct"]
    reasoning: str

router_llm = llm.with_structured_output(RouteDecision)
ROUTER_PROMPT = """You are a query router for a hybrid RAG system. Classify the query into ONE category: ... Query: {question}"""

def router_node(state: AgentState) -> AgentState:
    decision = router_llm.invoke(ROUTER_PROMPT.format(question=state["question"]))
    print(f"→ Router: {decision.route} | {decision.reasoning}")
    return {**state, "route": decision.route}

def route_edge(state: AgentState) -> str:
    return state["route"]

Step 3 – Retriever Nodes

# Vector retriever
def vector_node(state: AgentState) -> AgentState:
    docs = vector_retriever.invoke(state["question"])
    return {**state, "documents": docs}

# Graph retriever (Neo4j)
from langchain.chains import GraphCypherQAChain
graph_chain = GraphCypherQAChain.from_llm(llm=llm, graph=neo4j_graph, verbose=True, allow_dangerous_requests=True)

def graph_node(state: AgentState) -> AgentState:
    result = graph_chain.invoke(state["question"])
    doc = Document(page_content=result["result"], metadata={"source": "neo4j_graph"})
    return {**state, "documents": [doc]}

# Web retriever
def web_node(state: AgentState) -> AgentState:
    results = web_search.invoke(state["question"])
    docs = [Document(page_content=r["content"], metadata={"source": r["url"]}) for r in results]
    return {**state, "documents": docs}

# Direct LLM call
def direct_node(state: AgentState) -> AgentState:
    answer = llm.invoke(state["question"]).content
    return {**state, "generation": answer, "documents": []}

Step 4 – Grader, Rewriter, Generator, Hallucination Checker

# Grader
class GradeDoc(BaseModel):
    score: Literal["relevant", "irrelevant"]

grader_llm = llm.with_structured_output(GradeDoc)

def grader_node(state: AgentState) -> AgentState:
    grades = []
    for doc in state["documents"]:
        result = grader_llm.invoke(
            f"Question: {state['question']}
Document: {doc.page_content[:400]}
Is this document relevant to answering the question? Score: relevant/irrelevant"
        )
        grades.append(result.score)
    return {**state, "grade_results": grades}

def grade_edge(state: AgentState) -> str:
    relevant = sum(1 for g in state["grade_results"] if g == "relevant")
    if relevant > 0:
        return "generate"
    elif state["rewrite_count"] < 3:
        return "rewrite"
    else:
        return "web_fallback"

# Rewriter
def rewriter_node(state: AgentState) -> AgentState:
    rewritten = llm.invoke(
        f"The query '{state['question']}' returned no relevant results. Rewrite it to be more specific and searchable. Return only the rewritten query."
    ).content
    print(f"→ Rewriter: '{state['question']}' → '{rewritten}'")
    return {**state, "question": rewritten, "rewrite_count": state["rewrite_count"] + 1}

# Generator
def generator_node(state: AgentState) -> AgentState:
    context = "

".join(d.page_content for d in state["documents"] if "relevant" in state.get("grade_results", []))
    answer = llm.invoke(
        f"Answer using only the context below.

Context:
{context}

Question: {state['question']}"
    ).content
    return {**state, "generation": answer}

# Hallucination Checker
class HallucinationCheck(BaseModel):
    grounded: Literal["yes", "no"]

halluc_llm = llm.with_structured_output(HallucinationCheck)

def hallucination_node(state: AgentState) -> AgentState:
    context = "

".join(d.page_content for d in state["documents"])
    result = halluc_llm.invoke(
        f"Context:
{context}

Answer:
{state['generation']}

Is the answer fully supported by the context? grounded: yes/no"
    )
    return {**state, "hallucination_check": result.grounded}

def halluc_edge(state: AgentState) -> str:
    return "end" if state.get("hallucination_check") == "yes" else "regenerate"

Step 5 – Assemble the Graph and Run

workflow = StateGraph(AgentState)
workflow.add_node("router", router_node)
workflow.add_node("vector", vector_node)
workflow.add_node("graph", graph_node)
workflow.add_node("web", web_node)
workflow.add_node("direct", direct_node)
workflow.add_node("grader", grader_node)
workflow.add_node("rewriter", rewriter_node)
workflow.add_node("generator", generator_node)
workflow.add_node("hallucination", hallucination_node)

workflow.set_entry_point("router")
workflow.add_conditional_edges("router", route_edge, {"vector": "vector", "graph": "graph", "web": "web", "direct": "direct"})
for node in ["vector", "graph", "web"]:
    workflow.add_edge(node, "grader")
workflow.add_conditional_edges("grader", grade_edge, {"generate": "generator", "rewrite": "rewriter", "web_fallback": "web"})
workflow.add_edge("rewriter", "router")
workflow.add_edge("generator", "hallucination")
workflow.add_conditional_edges("hallucination", halluc_edge, {"end": END, "regenerate": "generator"})
workflow.add_edge("direct", END)

agent = workflow.compile()
result = agent.invoke({
    "question": "What is our data retention policy for EU customers?",
    "rewrite_count": 0,
    "documents": [],
    "grade_results": [],
    "generation": "",
    "route": "",
})
print(result["generation"])

Conclusion

By constructing a LangGraph state machine, the traditional one‑way RAG pipeline is upgraded to an intelligent agent with memory, decision‑making, and self‑correction. The router directs queries to the most suitable retriever (vector, graph, web, or direct), while the grader filters relevance, the rewriter repairs failed queries, and the hallucination checker ensures grounded answers, resulting in a robust, multi‑strategy RAG system.