Building a RAG Agent with LangGraph: Precisely Answer Your Private Knowledge Base

RAG Agent Architecture

┌─────────────────────────────────────────────────────────────┐
│              RAG Agent Workflow                           │
│                                                             │
│  User Question ──▶ Retrieval Tool ──▶ Get Context ──▶ LLM generates Answer │
│               │               │               │
│               ▼               ▼               ▼
│        Knowledge Base Docs   Source Citations          │
└─────────────────────────────────────────────────────────────┘

Core Components

VectorStore – stores document vectors.

Embeddings – converts text into vectors.

retrieve_documents – tool that fetches relevant documents from the knowledge base.

search_web – tool that performs an internet search for up‑to‑date information.

Agent – understands the query, schedules tools, and generates the final answer.

Code Implementation

from typing import TypedDict, Annotated
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.memory import InMemorySaver
import operator

class RAGState(TypedDict):
    question: str
    context: str
    answer: str
    sources: Annotated[list, operator.add]

@tool
def retrieve_documents(query: str) -> str:
    """Retrieve relevant docs from the knowledge base"""
    docs = [
        "LangGraph is a framework for building stateful, multi‑step LLM applications",
        "StateGraph is the core of LangGraph, defining workflows",
        "Checkpoint saves and restores state"
    ]
    relevant = [d for d in docs if any(k in d for k in query.split())]
    return "
".join(relevant) if relevant else "No relevant documents"

@tool
def search_web(query: str) -> str:
    """Search the internet for up‑to‑date information"""
    return f"Search '{query}' results..."

model = ChatOpenAI(model="gpt-4o")
agent = create_react_agent(
    model,
    [retrieve_documents, search_web],
    checkpointer=InMemorySaver()
)

result = agent.invoke({"messages": [{"role": "user", "content": "What is the core of LangGraph?"}]})

Continuous Dialogue Demo

config = {"configurable": {"thread_id": "rag-session-1"}}
# Round 1
result1 = agent.invoke({"messages": [{"role": "user", "content": "What is StateGraph?"}]}, config)
# Round 2 (context remembered automatically)
result2 = agent.invoke({"messages": [{"role": "user", "content": "How does it differ from Chain?"}]}, config)

Connecting a Real Vector Store

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

# Create vector store
vectorstore = Chroma(
    persist_directory="./chroma_db",
    embedding_function=OpenAIEmbeddings()
)

# Add documents
vectorstore.add_texts([
    "LangGraph is the underlying implementation of LangChain",
    "StateGraph provides a graph structure to organize workflows"
])

# Retrieval
docs = vectorstore.similarity_search("StateGraph", k=3)

Performance Comparison

Understanding Ability – Pure RAG: general; RAG Agent: strong (handles complex queries).

Continuous Dialogue – Pure RAG: not supported; RAG Agent: supported.

Tool Invocation – Pure RAG: single retrieval; RAG Agent: multiple tools callable.

Error‑Correction – Pure RAG: none; RAG Agent: self‑correcting.

Related Links

Official docs: https://langchain-ai.github.io/langgraph/
RAG guide: https://python.langchain.com/docs/tutorials/rag/