Building a Production-Ready RAG System: Architecture, Challenges, and Best Practices

Background and Motivation

Large language models (LLMs) face three major issues in real‑world applications: hallucinations, outdated knowledge, and data privacy risks. Retrieval‑Augmented Generation (RAG) addresses these problems by combining external retrieval systems with generative models, enabling up‑to‑date, verifiable, and context‑aware responses.

RAG Core Components

Data Sources: repositories of searchable information.

Data Processing: transforms raw data into RAG‑compatible formats.

Retriever: fetches relevant documents based on user queries.

Ranker: orders retrieved items for the LLM.

Generator: produces the final answer using the query and ranked knowledge.

Modular RAG Architecture

The system is organized into three layers:

Algorithm Layer : OCR, layout analysis, table recognition, and multi‑turn query rewriting.

Process Layer : offline indexing (document parsing, tokenization, vector creation) and online QA (query rewriting, hybrid retrieval, ranking). Underlying storage includes vector databases, Elasticsearch, and MySQL.

Configuration Layer : knowledge‑base management, model selection, and dialogue rules.

Offline Pipeline

Documents (PDF, Word) are ingested, OCR‑processed, layout‑reconstructed, and split into logical chunks. Each chunk is tokenized and embedded using selected vector models (BGE‑M3 and BCE) and indexed for both full‑text and vector search.

Online Pipeline

When a user query arrives, multi‑turn query rewriting (via TPLinker) resolves coreferences and fills missing information. Hybrid retrieval combines vector similarity and BM25 full‑text search, followed by a two‑stage ranking process:

Coarse Ranking (RRF fusion of multiple retrieval scores).

Fine Ranking using models such as ColBERT (token‑level interaction) and cross‑encoder re‑rankers.

A knowledge‑filtering step (NLI‑based binary classifier) removes irrelevant passages before generation.

Generation Strategy

Ranked knowledge is formatted into a prompt template with distinct knowledge and question sections. To improve answer fidelity, a two‑stage FoRAG approach first generates an outline and then expands it into the final response.

Key Lessons and Recommendations

Building a basic RAG system is straightforward; achieving production quality requires careful tuning of every component.

Prioritize comprehensive retrieval, robust ranking, and precise generation to reduce reliance on the LLM alone.

Hybrid retrieval (vector + BM25) and multi‑stage ranking balance recall and precision.

Knowledge filtering and prompt engineering are essential for factual accuracy.

Q&A Highlights

Launch criteria : manual evaluation of QA pairs, bad‑case resolution rate, and overall accuracy.

Handling incomplete context : supplement missing layers based on document hierarchy while respecting model token limits.

Improving latency : adopt lightweight rankers (e.g., ColBERT) or optimize hardware.

Optimizing beyond chunk size : ensure lossless parsing and appropriate chunk granularity.

Multimodal support : future integration of image and video processing.