RAG from Prototype to Production: Scenarios, Pitfalls, and Deployment Strategies

What Is RAG

Retrieval-Augmented Generation (RAG) lets a language model consult external documents when answering. The model handles understanding and expression; the retrieval system supplies candidate evidence. Adding a new document does not require retraining the generation model. The original paper also describes joint training of retriever and generator, but application-layer RAG can be simpler.

Two main pipelines exist:

Document preparation: Parse PDFs, Word files, web pages; split by paragraph, section, or other logical boundaries; build a retrieval index. Each chunk retains title, source, and version for later identification and citation.

Answering questions: On user query, retrieve candidate chunks, filter and organize context, then call the model to generate an answer. Document preparation is asynchronous; each question does not reprocess all documents.

Vector search is one retrieval method: embedding models convert text to vectors, using distance or similarity to capture semantic relations. Keyword search excels at matching exact names, IDs, and terminology. Both can be combined; RAG can also use databases, search engines, or other tools. Choosing a vector database only decides one implementation detail.

RAG supplements the model with enterprise knowledge and provides a path to update knowledge and show sources. However, retrieved candidates may be wrong, and the model may misread. Sources make answers easier to verify, but correctness still requires separate evaluation.

Suitable Scenarios for RAG

First examine what information the question depends on. In a reimbursement assistant:

"How much can I claim for accommodation?" needs applicable policy — suitable for RAG to find basis and explain rules.

"Where is my reimbursement in the process?" needs real-time business state — should query the reimbursement system.

"Help me submit this reimbursement." needs an action with side effects — should use business APIs for authorization, validation, and submission.

RAG is worth evaluating first when answers depend on large external text corpora, users need explanations, summaries, or citations, and the material cannot fit entirely into each request's context. Examples: enterprise policy Q&A, product manual lookup, technical support, cross-document verification, and providing evidence location for human reviewers.

Comparison of approaches for different tasks:

Small, fixed document sets: Compare direct context injection vs. RAG — evaluate answer quality, call cost, and update mechanism.

Large document collections: Compare RAG vs. existing search — evaluate retrieval coverage, provenance, and user task completion efficiency.

State queries and statistics: Use SQL or business APIs — rely on verifiable current data and calculation rules.

Submissions or modifications: Use controlled tool calls — let business services handle authorization, validation, and execution results.

When documents are few and scope fixed, putting them directly into context may be simpler. When documents are many, update frequently, or require per-user filtering, on-demand retrieval is worth comparing. Selection should be based on results on the same set of real questions, not on a predetermined decision to build a vector index.

Fine-tuning typically improves model behavior, format, and specific task capabilities. Continuously changing policies, real-time state, and access permissions still need independent data sources and control mechanisms. A complete assistant can combine these capabilities, but each must have a clear responsibility.

Six Common Production Pitfalls

1. Documents Visible but Content Not Correctly Read

Tables, scanned pages, footnotes, and headers in PDFs may be lost or misaligned during parsing. Humans see "East China region limit 500 yuan"; the system may only read "500 yuan", losing the region–amount association. Spot-check parsing results, especially tables, conditions, and exceptions, and retain location information to trace back to the original text.

2. Chunking Breaks Links Between Rules and Exceptions

Fixed-length chunking can separate "usually reimbursable" from the next paragraph's "except in the following cases". If retrieval only returns the first chunk, the model may miss the restriction. Improvements include preserving section paths, splitting on semantic boundaries, adding necessary context, and retrieving adjacent related chunks. Chunk size and overlap should be tuned with real question tests. [2]

3. Semantically Similar Documents Have Different Applicability

Old policies, new policies, regional variants, and employee-type variants may have high semantic similarity. Retrieval scores cannot substitute for business applicability judgment. Version, effective date, region, and applicable audience must become explicit data fields; historical questions should be retrieved against the corresponding time version, not always the latest.

4. Multi-Condition Questions Retrieve Only Partial Evidence

"Off-site business trip with last-minute rebooking — what expenses are claimable?" may involve travel policy, approval requirements, and rebooking rules. A single retrieval pass that fetches one relevant rule does not guarantee complete evidence. Split the question into multiple retrievals, then merge evidence; as complexity grows, limit retrieval rounds and call budget, and verify whether quality actually improves.

5. Citations Exist but Conclusions Lack Support

The model may cite a relevant passage yet add amounts, scopes, or exceptions not present in the source. Validation must check both provenance and key conclusions: does this excerpt explicitly support the judgment, and are conditions preserved? Low similarity scores cannot be directly mapped to answer confidence. Critical conclusions need verifiable evidence, with human review when necessary.

6. Permissions and Caching Cause Data Leakage to Wrong Users

Once retrieved content enters the model context, it has crossed the data access boundary. Permission filtering must be enforced server-side; identity and authorization scope come from trusted authentication results. Caching must account for tenant, permission scope, and document version; the same question from different users may not reuse the same answer.

These issues compound along the pipeline. A 2024 RAG engineering experience report discusses failure points in three domain cases, showing that both retrieval and generation need continuous verification. [3] For a specific project, a more effective practice is to log which documents were used for each answer, at which step the error occurred, then decide whether to fix parsing, retrieval, or generation.

Modules Required for a Production RAG System

A production solution can be built across three layers: document pipeline, QA pipeline, and operational support. Start with one well-defined scenario; choose components based on data scale, concurrency, and update requirements; do not introduce all middleware upfront for architectural completeness.

Document Management Handles Full Lifecycle

Each document must at minimum identify source, document ID, version, applicability scope, and access permissions; chunks retain parent section and original position. These fields serve retrieval filtering, answer citation, and debugging. Auto-generated summaries or supplementary notes must be distinguished from original text to avoid treating model-generated explanations as official policy.

Update flow: receive documents → parse and validate → build candidate index → test → publish. On parse failure, retain failure state; do not silently fall back to old documents. When replacing or deleting documents, synchronize removal of old chunks and caches to prevent revoked content from being used in answers.

Permission revocation needs an independent effective path. While index async update is pending, the service must still restrict reads based on current authorization; it cannot wait for index rebuild to stop access. Different projects may implement differently, but must define how long updates take to become visible and who handles failures.

QA Entry Establishes Identity and Task First

The server first establishes user identity and data scope, then decides whether this turn needs document lookup, real-time state query, or action execution. A user's claim "I am an admin" in text cannot change authorization. When region, time, or business object are missing and affect the conclusion, the assistant should ask for clarification rather than assume.

Composite requests can be split into controlled steps. Example: "Check the policy, then submit this reimbursement" — first retrieve and explain rules, then let the business service validate order and permissions; show retrieval success and submission success separately. Retrieval results can assist decisions, but actual submission results must come from the business system.

Retrieval Considers Both Matching and Applicability

For queries containing IDs, product names, or domain terms, evaluate hybrid keyword + vector retrieval. Retrieve candidates, then deduplicate and filter; a reranker can further judge relevance but adds cost and latency. Candidate count, chunking strategy, and whether to rerank should all be compared on the same evaluation set. [2]

Identity permissions and business scope must enter as retrieval constraints, not left to the generation model. Microsoft's security filter example filters results using user or group identifiers stored in documents, while clarifying that the filter field itself does not perform authentication. [4] The application must still ensure each query's authorization conditions come from a trusted server and cover caching, reranking, and result return.

Context Organization Preserves Evidence Relationships

Content fed to the model should include the question, necessary business conditions, candidate chunks, and explicit source identifiers. Rules and exceptions, tables and explanations must be organized together; irrelevant duplicates consume context budget. When sources conflict, show the conflict or apply a predetermined version rule; do not let the model guess authority based on phrasing.

Documents may contain text that tries to induce the model to ignore requirements, leak information, or call tools. Retrieved content must be treated as data for analysis, not granted tool permissions. Operations are still executed server-side with identity, parameter, and business rule validation; they cannot rely solely on a prompt to prevent privilege escalation.

Answers Have Explicit Success and Failure Outcomes

Ordinary explanations can return answer, provenance, and applicability scope; critical amounts, dates, and clause numbers must be cross-checked against evidence. Automated validation and model-based review can assist screening, but important conclusions still need human spot-checks. Correct data format only means output conforms to interface requirements.

When evidence is insufficient, state missing basis; when conditions are incomplete, ask; when sources conflict, show the conflict. Retrieval timeout and model timeout are technical failures — return them as such, optionally falling back to searchable original text or human handoff. How to proceed on failure should be defined by the product contract; unfounded free-form answers must not be the default fallback.

Operational Support Enables Debugging and Recovery

Each request logs a correlatable request ID, retrieved chunk IDs, data and model versions, processing latency, and error category. Log scope must balance debugging needs, data permissions, and retention requirements. This distinguishes "document not published", "retrieval missed recall", and "model misread".

Cache keys must cover identity scope, business conditions, and knowledge version that affect results. Invalidate on document or permission changes. Before launch, define timeout budgets, concurrency limits, dependency failure fallback modes, and rollback paths for index and configuration. Capacity and cost are measured by real load tests; component count cannot replace stress-test results.

Proving Business Requirements Are Met

Build an evaluation set first, then tune parameters. Questions must come from real business, covering common phrasings, long-tail questions, unanswerable questions, missing conditions, version conflicts, cross-document verification, and unauthorized requests. Each item records applicability scope, key evidence, acceptable answers, and expected failure modes, reviewed by domain experts.

The same question set must compare meaningful baselines: current manual search, standard search, direct context injection, and RAG. Keep debugging set separate from final acceptance set to avoid overfitting to repeatedly tuned questions. Microsoft's design guide also recommends phased evaluation, with final user experience determining acceptance criteria. [5]

Acceptance dimensions:

Documents & Retrieval: Are correct-version key evidences complete? — Spot-check originals and evidence recall.

Answers & Citations: Is the answer correct and supported by sources? — Business review and citation verification.

Follow-up & Refusal: Can it answer when it should, and refuse when evidence is lacking? — Separate statistics for answerable and unanswerable samples.

Permissions & Updates: Are unauthorized requests isolated? Do changes take effect promptly? — Test with different identities and update/revocation scenarios.

Business & Operations: Do users complete tasks? Are cost and latency acceptable? — Task logs and end-to-end runtime data.

Track retrieval recall, answer accuracy, and business task completion rate separately. Also monitor effective answer coverage and appropriateness of refusals to avoid inflating accuracy by excessive refusals. Reports must state sample size, error types, and test environment; averages must not hide unauthorized access or outdated policy answers.

Failed samples are diagnosed across four layers: documents, retrieval, answers, business. Example: if the rule is in candidates but the answer misses an exception, prioritize checking context organization and generation; if the rule was never retrieved, then check indexing, filtering, and recall. Every adjustment re-evaluates benefit vs. cost; avoid judging improvement based on a few demo questions.

From Limited Trial to Continuous Operation

Phase 1: Pick a task with clear boundaries, e.g., travel policy lookup. Define document scope, target users, and success criteria. Let users open sources directly; have domain experts verify key answers. Focus on validating parsing, versioning, and retrieval to quickly uncover data gaps.

Phase 2: Add permissions, update/delete handling, follow-up/refusal logic, failure fallback, and request tracing; verify real dependencies in target environment. Test both normal answers and scenarios like user permission changes, document revocation, dependency timeouts, and service restarts. Recovery capability must be validated against actual state and results.

Phase 3: Limited user pilot. Record human corrections, unresolved issues, cost, and end-to-end latency. Feed new failure samples into regression evaluation, then gradually expand scope. Each release pins data, index, model, and config versions, with explicit rollback triggers. Rollback must also respect current permissions — cannot restore already revoked access.

For the reimbursement assistant: step one finds applicable policy with accurate citations; step two connects reimbursement status query; submission is handled separately with authorization and business validation. Each step has verifiable outcomes so the team can judge whether adding a capability truly helps users complete work.

RAG's value ultimately shows in these concrete results: users find applicable evidence faster, answers are verifiable, document changes take effect promptly, and errors are traceable and handleable. Production design and acceptance must revolve around these outcomes.

References

[1] Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks , 2020.

[2] Anthropic, Introducing Contextual Retrieval , 2024.

[3] Scott Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System , 2024.

[4] Microsoft Learn, Security filters for trimming results in Azure AI Search .

[5] Microsoft Learn, Design and develop a RAG solution .