How RAG Can Overcome Large‑Model Pitfalls in Enterprise Knowledge Work

As large language models (LLMs) rapidly evolve, the gap between Chinese models and OpenAI’s offerings narrows, yet practical deployment still encounters major issues such as hallucinations, outdated knowledge, and data privacy risks. Retrieval‑Augmented Generation (RAG) addresses these problems by combining external retrieval systems with generative models, allowing up‑to‑date, verifiable, and context‑aware responses.

Background and Motivation

Hallucination: models fabricate answers when no factual data exists.

Staleness: knowledge updates are slow, costly, and often outdated.

Security & privacy: risk of data leakage or misuse.

RAG mitigates these issues by grounding generation in retrieved documents.

RAG Core Architecture

The modular RAG design consists of several layers:

Data sources : repositories of searchable content (PDFs, Docs, databases).

Data processing : OCR, layout analysis, table extraction, and chunking to create knowledge blocks.

Retriever : fetches relevant chunks using hybrid retrieval (vector + BM25 + knowledge graph).

Ranker : applies coarse ranking (RRF) followed by fine‑grained ranking (ColBERT, cross‑encoder) and optional knowledge‑filtering via NLI.

Generator : combines the user query with the top‑ranked knowledge blocks using a prompt template, then generates the final answer.

Images illustrate the traditional RAG pipeline, the modular extension, and the end‑to‑end workflow.

Implementation Details

Document Parsing

Documents are processed with the open‑source RAGFlow DeepDoc module, which handles PDF OCR, layout recovery, table recognition, and Word structure preservation. After parsing, content is split into two levels of chunks: structural (title, subtitle, body) and length‑based (e.g., 256‑token slices). This balances retrieval recall and generation fidelity.

Query Rewriting

Multi‑turn queries are reformulated as a relation‑extraction task using the TPLinker model, treating coreference resolution and information completion as entity relations.

Hybrid Retrieval

Two parallel retrieval streams are employed:

Vector search (e.g., BGE‑M3, BCE) for semantic similarity and multilingual support.

Full‑text BM25 search for exact keyword matching.

Results from both streams are merged using Reciprocal Rank Fusion (RRF), which relies on relative rankings rather than raw scores, providing a simple yet effective fusion.

Ranking Strategy

A two‑stage ranking pipeline is used:

Coarse ranking (RRF) reduces 100 candidates to the top 20.

Fine ranking applies ColBERT (delayed interaction token‑level model) or a cross‑encoder for precise relevance scoring, followed by knowledge filtering via an NLI classifier.

This combination improves both efficiency and accuracy, especially for large candidate sets.

Answer Generation

After ranking, knowledge blocks are formatted (knowledge layout) and inserted into a prompt template with placeholders for knowledge and question. To enhance factual consistency, a two‑stage generation (FoRAG) first produces an outline, then expands it into a full answer.

Practical Insights and Challenges

Chunk size trade‑off: smaller chunks improve retrieval precision but may lose context; larger chunks preserve semantics but risk irrelevant content.

Multimodal data (audio/video) is not yet supported; future work includes multimodal extensions.

Table‑heavy PDFs require full‑table ingestion; precise region extraction would further boost accuracy.

System latency can be reduced by selecting lightweight ranking models (e.g., ColBERT) when hardware is limited.

Evaluation and Deployment

Performance is measured by offline relevance tests (BGE‑M3 vs. BCE) and online metrics such as bad‑case resolution rate and overall answer accuracy. Continuous feedback from enterprise users guides iterative improvements.

Conclusion

Building a production‑grade RAG system involves careful engineering across data ingestion, query rewriting, hybrid retrieval, multi‑stage ranking, and prompt‑driven generation. Success requires balancing model capabilities with domain‑specific optimizations rather than relying solely on LLM power.