What Is Retrieval‑Augmented Generation (RAG) and Why Must Large Models Look Up Information First?

What is RAG?

Retrieval‑Augmented Generation (RAG) means the model first retrieves relevant documents and then generates an answer, similar to an open‑book exam. The model alone only knows what it saw during training, so feeding it external knowledge enables reliable answers.

Why is RAG needed?

Large models cannot answer company‑specific questions such as internal travel reimbursement policies because they have never seen those private documents. Without retrieval the model would hallucinate. RAG solves this by precisely feeding the most relevant knowledge to the model.

Full RAG workflow

1. Build the index

Document parsing – convert PDFs, Word files, web pages, etc., into plain text.

Text chunking – split long texts into smaller “chunks” so that retrieval can locate relevant pieces quickly.

Text vectorization – use an embedding model to turn each chunk into a numeric vector; semantically similar sentences obtain nearby vectors.

Store the vectors in a vector database for fast similarity search.

2. Retrieve and generate

Retrieval – encode the user query into a vector, search the vector database for the most similar chunks, optionally apply reranking, hybrid search, or query rewriting to improve relevance.

Generation – assemble the retrieved chunks and the original question into a prompt such as “Please answer the user’s question based on the following information: {retrieved chunks}. Question: {user query}.” The large model then performs reading‑comprehension and summarization rather than relying on its internal knowledge.

Key practical considerations

The overall process sounds simple, but engineering details—how to split documents, which embedding model to choose, retrieval strategy, whether to rerank results, and how to craft the prompt—significantly affect the final answer quality. These topics will be explored in future posts.