How RAG Can Stop AI Hallucinations: A Hands‑On Guide

During a demo for a company‑wide customer‑service chatbot, the author fed product documentation to a large language model and observed impressive answers at first, but when asked a question not covered by the docs the model confidently gave a completely wrong response—illustrating the classic hallucination problem of LLMs.

Retrieval‑Augmented Generation (RAG) is introduced as the remedy: the system first retrieves relevant passages from a knowledge base, injects those passages into the prompt, and then lets the model generate an answer grounded in the retrieved material.

The most overlooked step is data preprocessing. The author stresses that before vectorisation every document must be cleaned—remove headers/footers, strip garbled characters, convert tables to structured text, and ensure a uniform format (PDF, Word, or HTML). This tedious work sets the upper bound for all downstream stages.

Chunking is another critical decision. Documents should be split into chunks of a few hundred tokens and stored as separate vectors. Three strategies are compared:

Fixed‑length chunks: simple but often cuts concepts in half, leading to incoherent retrieval.

Semantic chunks: higher quality but requires more complex tooling.

Overlapping paragraph chunks: the author’s preferred compromise, keeping about 50 tokens of overlap between adjacent chunks so that context is preserved even when cut points are imperfect.

For the vector store, many options exist—Faiss, Chroma, Weaviate, Milvus, etc. The recommendation is to use Chroma for local development because it starts with a few lines of code, and switch to Milvus or a cloud‑hosted service when production data volumes grow.

The core operations are straightforward: during ingestion, convert each cleaned chunk into an embedding vector and write it to the database; during query, embed the user question, perform a nearest‑neighbor search, and retrieve the top‑k (e.g., 5) passages.

Frameworks such as LangChain or LlamaIndex encapsulate these steps, allowing developers to focus on the workflow rather than low‑level retrieval code. The author advises first getting a working pipeline with a framework before digging into the underlying mechanics.

Retrieval quality still determines answer quality. The author lists three tuning knobs:

Adjust the similarity threshold and discard passages below a certain score.

Apply a rerank model ( rerank) to reorder the retrieved snippets; this often improves answer quality more than switching to a larger base model.

Explicitly instruct the LLM in the prompt: “Answer only based on the following content; if the content does not contain the answer, say you don’t know.” This constraint curbs the model’s tendency to hallucinate.

A minimal runnable pipeline is outlined:

Load documents with LangChain.

Chunk them using the overlapping‑paragraph method.

Embed each chunk via the OpenAI embedding API.

Store embeddings in Chroma.

When a user asks a question, retrieve the top‑5 most similar chunks.

Append those chunks to the prompt and invoke GPT‑4 to generate the answer.

The entire flow can be implemented in a few hundred lines of code. After it runs, the author notes that most failures stem from poor document quality, inappropriate chunking, or irrelevant retrieval results—not from the model itself. RAG effectively bounds the model’s knowledge to the supplied data, reducing hallucinations to an acceptable level for enterprise knowledge bases, customer‑service Q&A, and internal document search.

Finally, the process is iterative: continuously refine the data cleaning, adjust chunking parameters, tune similarity thresholds, and retrain rerankers to keep the system performant.