Building an Enterprise‑Grade RAG 2.0 System: Architecture, Challenges, and Best Practices

Background

Large language models (LLMs) encounter hallucination, freshness, and data‑privacy issues when applied to enterprise scenarios. Retrieval‑Augmented Generation (RAG) mitigates these problems by combining external knowledge retrieval with LLM generation.

RAG Core Architecture

Traditional RAG consists of indexing, retrieval, and generation. The modular RAG design adds pre‑retrieval query rewrite (HyDE) and post‑retrieval rerank, filtering, and routing components.

System Design for Enterprise

The system is organized in three layers:

Algorithm components (OCR, multi‑turn query rewrite, tokenization, table recognition).

Workflow construction (offline ingestion and online Q&A).

User‑level configuration (knowledge‑base management, model selection, dialogue rules).

Offline Pipeline

Document parsing (OCR, layout recovery, table recognition) creates logical blocks.

Chunking balances length (e.g., 256 tokens) to avoid loss of context.

Text tokenization and vector embedding using BGE‑M3 and BCE are written to both text and vector indexes.

Online Pipeline

Multi‑turn query rewrite modeled as a relation‑extraction task with TPLinker.

Hybrid retrieval combines vector search (semantic similarity) and BM25 full‑text search.

Two‑stage ranking: coarse RRF fusion selects top‑20 from 100 candidates, then fine‑grained reranker (ColBERT or Cross‑Encoder) picks top‑5.

Knowledge filtering via an NLI‑based binary classifier ensures relevance.

Generation

After ranking, knowledge blocks are formatted and inserted into a prompt template (knowledge + question). A two‑stage FoRAG generation first produces an outline, then expands it to a final answer, improving structure and completeness.

Model Comparisons and Choices

Segmentation models: jieba and LAC are overly fine‑grained; texsmart is too coarse; cutword offers a balanced granularity.

Vector models evaluated: BGE‑M3 and BCE were selected for complementary performance; newer models were tested but not adopted.

Ranking models: RRF (non‑model fusion), ColBERT (late interaction, token‑level), and Cross‑Encoder (interactive, higher latency) are used according to efficiency‑accuracy trade‑offs.

Evaluation and Lessons

Key metrics focus on three goals: “search more fully” (retrieval completeness), “rank better” (ordering relevance), and “answer more precisely” (generation accuracy). The design emphasizes modularity, horizontal scalability, plug‑in algorithms, low cost, and maintainability.

Q&A Highlights

Launch criteria: manual evaluation of bad‑case resolution rate and overall accuracy.

Context completion strategies for missing hierarchical information.

Latency mitigation by selecting lightweight ranking models such as ColBERT.

Chunk size and document parsing impact retrieval and ranking performance.

Future plans include multimodal support for audio‑video content.