RAG Basics: Indexing and Query Phases Explained for Interviews

What is RAG?

Retrieval‑Augmented Generation (RAG) combines a retrieval step with a large language model generation step, allowing the model to consult external documents before answering a user query.

Two‑Stage Workflow

RAG consists of an offline indexing stage and an online query stage that are executed at different times.

Indexing Stage (offline)

Document chunking : Original texts are split into semantically or length‑appropriate fragments. Too large fragments reduce retrieval precision; too small fragments lose context.

Embedding vectorization : Each fragment is encoded into a fixed‑dimensional vector using an embedding model (e.g., bge-large-zh ). Semantically similar texts obtain similar vectors.

Vector storage : Vectors together with their source texts are stored in a vector database such as Milvus or Faiss, ready for later retrieval.

Query Stage (online, triggered by a user question)

Question embedding : The same embedding model converts the user’s question into a vector.

Similarity search : The vector database performs cosine‑similarity search to retrieve the top‑K most relevant document fragments.

Augmented generation : The retrieved fragments are concatenated with the original question to form a prompt, which is then fed to a large language model that generates an answer grounded in the retrieved material.

Key Bottleneck and Tuning Parameters

Retrieval accuracy is the core bottleneck—irrelevant documents lead to garbage‑in‑garbage‑out generation. Two parameters strongly affect performance:

Chunk size : 256–512 tokens work well; overlapping chunks prevent semantic truncation.

topK : Number of retrieved results, typically 10–20, tuned experimentally.

Real‑World Scenarios

Internal knowledge bases : Company policies and technical wikis are vectorized; employees can ask natural‑language questions like “How to apply for annual leave?” and receive precise answers.

Product documentation assistants : SDK docs and API manuals are embedded; developers query “How to call the payment API?” and get code‑sample enriched responses.

Education / exam prep : Textbooks and notes are vectorized; students ask “What is gradient descent?” and receive synthesized explanations from multiple sources.

Common interview question: “What strategies exist for document chunking in the indexing stage?” Answer: Fixed‑length splitting, semantic splitting at sentence or paragraph boundaries, and paragraph‑level splitting; chunk size of a few hundred tokens with overlap is usually optimal.

Mnemonic: RAG = Indexing (Chunk → Embed → Store) + Query (Embed → Retrieve → Generate); retrieval accuracy is the critical bottleneck.