RAG Explained: Core Concepts, Architecture, and a Minimal LangChain Demo

1. RAG Origin

RAG (Retrieval-Augmented Generation) was introduced in the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" [1]. The core idea combines parametric memory (a pre-trained sequence-to-sequence generator) with non-parametric memory (a dense retrieval index over an external knowledge base such as Wikipedia) to improve performance on knowledge-intensive NLP tasks.

RAG consists of two main components:

Retriever : Based on a pre-trained neural retrieval model (the paper uses Dense Passage Retriever), it fetches the most relevant text passages from a large document index given an input question or context.

Generator : Based on a pre-trained seq2seq model (the paper uses BART-large), it fuses the retrieved passages with the original input to produce the final answer.

Three key advantages:

Interpretability and verifiability : Generated output can be traced back to specific retrieved documents, providing provenance and transparency.

Updatable knowledge : Updating the external knowledge base (non-parametric memory) instantly equips the model with new information without retraining the entire LLM.

Reduced hallucination : Generation is constrained by real retrieved text, yielding more specific, diverse, and factually grounded content.

2. Simple RAG Architecture

The embedding-based RAG architecture comprises two pipelines:

Offline data ingestion pipeline (dashed lines in the diagram).

Online retrieval-generation pipeline (solid lines).

While the high-level flow is straightforward, each stage presents engineering challenges.

2.1 Chunk Size

How to split documents and at what granularity has no definitive answer.

Smaller chunks fit more passages into the model's limited context window, providing broader information coverage for more comprehensive answers.

Overly small chunks risk fragmenting or losing key information : if a topic's complete information is scattered across multiple chunks, individual chunks may lack sufficient relevance to be retrieved, causing critical details to be missed.

Excessively small chunks also significantly increase compute and storage costs : more chunks mean more embeddings to generate, store, and search, expanding the vector search space and potentially slowing queries.

2.2 Retrieval Strategies

Vector Search Algorithms

Vector search is typically framed as approximate nearest neighbor (ANN) search. The naive k-NN is too slow for large datasets. Mainstream ANN algorithms include:

LSH (Locality-Sensitive Hashing) : Hashes similar vectors into the same buckets for fast similarity search with slight precision loss. Supported by faiss [2] and Annoy [3].

HNSW (Hierarchical Navigable Small World) : Builds a multi-layer graph where nodes are vectors and edges connect similar vectors. Enables fast search and efficient incremental indexing. Implemented in faiss and Milvus [4].

Product Quantization (PQ) : Decomposes high-dimensional vectors into sub-vectors and compresses them, computing distances in the low-dimensional space. Core component of FAISS and adopted by most mainstream vector retrieval libraries.

IVF (Inverted File Index) : Uses K-means clustering to group similar vectors into clusters (typically 100–10,000 vectors per cluster). Queries first locate the nearest cluster centroid, then search within that cluster. Often combined with PQ, forming FAISS's primary retrieval framework.

Annoy (Approximate Nearest Neighbors Oh Yeah) : Builds multiple random binary partition trees; each tree splits the vector space via random hyperplanes. Retrieval traverses multiple trees and aggregates candidates.

Reranking

Beyond algorithmic improvements, reranking is an effective engineering optimization: a low-cost, lower-precision retriever first fetches a candidate list, then a higher-precision (but costlier) model reorders the candidates to select the best matches.

Contextual Retrieval

Contextual Retrieval [5], proposed by Anthropic (https://www.anthropic.com/engineering/contextual-retrieval), enhances retrievability by augmenting each chunk with relevant contextual information during preprocessing. According to Anthropic, this method reduces retrieval failure rate by 49%, and up to 67% when combined with reranking.

3. Minimal Example with LangChain

The author demonstrates a minimal RAG implementation using LangChain [6].

3.1 Building the Vector Database

LangChain abstracts text data as Document objects, supporting loaders for PDF, CSV, web, etc. The example loads a PDF book "Think.Distributed.Systems.2025.8.pdf" via PyPDFLoader, yielding a list<Document>.

For chunking, RecursiveCharacterTextSplitter is used, which tries to keep paragraphs, then sentences, then words together because these units have the tightest semantic cohesion.

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
)
chunks = splitter.split_documents(documents)

Chunk size is set to 1000 characters with 200-character overlap to mitigate information loss when context spans chunk boundaries. (Parameter values follow official documentation examples, not rigorous tuning.)

Generating Embeddings

The author uses the Hugging Face SentenceTransformer [7] framework with the all-MiniLM-L6-v2 model.

model = SentenceTransformer(
    "./model/all-MiniLM-L6-v2"
)
texts = [chunk.page_content for chunk in chunks]
embeddings = model.encode(texts, show_progress_bar=True)

Storing Embeddings

Vectors are stored in FAISS. Since the index only holds embeddings, the original chunk text is saved in a parallel metadata list for later lookup by index.

metadatas = [{"text": chunk.page_content} for chunk in chunks]
dim = embeddings.shape[1]
index = faiss.IndexFlatL2(dim)
index.add(np.array(embeddings).astype("float32"))

faiss.write_index(index, faiss_path)
pickle.dump(metadatas, f)

3.2 Retrieval and Generation

Querying Similar Vectors

The query is embedded, then FAISS returns the top-k nearest neighbor indices and distances.

query_emb = model.encode([query_text]).astype("float32")
D, I = index.search(query_emb, top_k)
results = []
for idx, dist in zip(I[0], D[0]):
    meta = self.metadata[idx] if idx < len(self.metadata) else None
    results.append({"index": idx, "distance": dist, "metadata": meta})

Assembling Prompt and Calling LLM

Retrieved chunk texts are concatenated into a context block, combined with the user query, and sent to the LLM (DeepSeek Chat).

query = "What is the challenge of reaching agreement?"
combined_context = "

".join([res["metadata"]["text"] for res in results if res["metadata"]])
prompt = f"Using the following context:
{combined_context}

Answer the question: {query}"
llm = ChatDeepSeek(api_key="sk-xx", model_name="deepseek-chat")
response = llm.invoke(prompt)

The model returned a satisfactory answer:

Based on the provided context, the challenge of reaching agreement in distributed consensus lies in the realistic system model, as opposed to a theoretical one. Specifically: In a theoretical model , where components do not fail and the network is perfectly reliable (delivers messages exactly once and in order), consensus is trivial to achieve. The challenge arises in a realistic model , where: Components may fail. The network may reorder, delay, lose, or duplicate messages. This combination of potential process failures and an unreliable network makes designing a consensus algorithm notoriously difficult. Simple solutions, like appointing a single leader (BDFL), are insufficient because if that leader fails, the system halts, and network problems can still prevent decisions from being consistently communicated. Answer: The challenge of reaching agreement is achieving it in a realistic system where processes can fail and the network is unreliable (can lose, duplicate, delay, or reorder messages), as opposed to a perfect theoretical system where consensus is easy.

4. Summary

RAG is conceptually simple to understand.

LangChain makes it easy to assemble a working RAG prototype.

The example code is available at https://gitee.com/shichaoyuan/first-rag-demo [8].

References

https://arxiv.org/abs/2005.11401 — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

https://github.com/facebookresearch/faiss — FAISS

https://github.com/spotify/annoy — Annoy

https://milvus.io/zh — Milvus

https://www.anthropic.com/engineering/contextual-retrieval — Contextual Retrieval (Anthropic)

https://www.langchain.com/ — LangChain

https://huggingface.co/sentence-transformers — SentenceTransformer

https://gitee.com/shichaoyuan/first-rag-demo — first-rag-demo