What Is Retrieval‑Augmented Generation (RAG) and Why It Matters for LLM Interviews

RAG Overview

Retrieval‑Augmented Generation (RAG) inserts an external knowledge base into the inference pipeline of a large language model (LLM). Before generating an answer, the model first retrieves relevant documents, then conditions its generation on those documents.

Why RAG is needed

Hallucination : LLMs predict the next token without factual grounding, which can produce confident but false statements. Example: using GPT‑3.5 for customer‑service led to recommendations of non‑existent products.

Knowledge cutoff : Training data is frozen at a certain date, so the model cannot answer questions about events after that date. A continuously updated knowledge base removes this temporal limitation.

Domain knowledge gap : Public internet corpora do not contain proprietary documents, product manuals, or internal FAQs. Injecting private corpora via RAG fills this gap.

Traceability : Pure LLM answers lack source citations. Retrieval provides explicit document references, enabling verification.

Core RAG workflow

Offline (knowledge‑base construction) :

Parse source files into plain text.

Split text into appropriately sized chunks.

Encode each chunk with an embedding model.

Store the resulting vectors in a vector database.

Online (query & generation) :

Encode the user query into an embedding.

Perform similarity search in the vector store to retrieve top‑k chunks.

Optionally rerank the retrieved set with a cross‑encoder for higher precision.

Assemble the selected chunks into a prompt.

Pass the prompt to the LLM to generate the final answer.

RAG vs. Fine‑tuning vs. Long‑context

Core role : RAG injects external knowledge; fine‑tuning changes model behavior/style; long‑context expands the input window.

Knowledge update : RAG updates the knowledge base at any time; fine‑tuning requires retraining; long‑context updates by changing the supplied input.

Cost : RAG incurs low compute cost (vector search + LLM inference); fine‑tuning is medium‑high (training compute + data preparation); long‑context costs scale with token usage.

Hallucination control : Retrieval constraints in RAG give better hallucination mitigation; fine‑tuning offers average control; long‑context depends on the richness of the supplied context.

Typical scenarios : RAG for knowledge‑intensive Q&A and enterprise KB; fine‑tuning for output format or style customization; long‑context for full‑document analysis or summarization.

Response latency : RAG adds a modest retrieval step; fine‑tuning adds no extra latency; long‑context latency grows with input length.

Typical stack : LangChain + Chroma for RAG; LoRA / QLoRA for fine‑tuning; GPT‑4o (128 K tokens) or Gemini (1 M+ tokens) for long‑context.

Emerging trends (2025‑2026)

Agentic RAG : The model decides autonomously whether to retrieve, what to retrieve, and whether to iterate retrieval, turning retrieval into an active decision rather than a fixed pipeline.

Graph‑RAG : Combines dense vector search with knowledge‑graph reasoning to handle entity‑relationship queries. Microsoft’s open‑source GraphRAG framework demonstrates this approach.

Context Engineering : Shifts focus from “how to retrieve” to “how to construct the optimal context” – selecting, ordering, compressing, and de‑conflicting retrieved chunks for maximal relevance.

Common interview follow‑up questions

How to improve poor retrieval? Optimize chunking (semantic splitting vs. fixed length), add a reranker, employ hybrid search (dense + sparse), rewrite the user query for better lexical match, and upgrade the embedding model.

When to use RAG vs. fine‑tuning? Can they be combined? Use RAG for rapidly changing factual knowledge; use fine‑tuning for altering model behavior or style. The two techniques are complementary and can be applied together.

Is RAG still necessary with very large context windows (e.g., Gemini 1 M tokens)? Yes. RAG remains cheaper (no need to embed entire corpora), offers lower latency by retrieving only a few relevant snippets, and provides traceable citations—advantages that large context windows alone do not guarantee.