How to Decompose a Production‑Ready RAG System for Interview Success

Answer in 30 seconds

I would split the RAG system into an offline ingestion chain and an online query chain. The offline chain handles file storage, format routing, parsing, cleaning, chunking, embedding, bulk indexing, validation, and version release, with task status and recoverable boundaries at each step. The online chain starts with identity and tenant scope, then performs query processing, permission filtering, keyword and vector retrieval, fusion, rerank, evidence thresholding, context construction, and streaming generation. A relational database stores users, documents, versions, and task status; the object store keeps raw files; the search engine stores chunks, vectors, and metadata; caches hold recomputable data. Failures trigger module‑level retries or graceful degradation, and a Trace links queries, candidates, models, and citations back to bad cases for regression.

Why offline ingestion and online query must be separate

Offline ingestion aims for completeness, recoverability, and traceability. It must handle diverse formats, OCR, tables, cross‑page content, chunking, embedding, and bulk writes, and it needs to know where a failure occurred and whether partial results must be cleaned.

Online query prioritises low latency, correct permissions, and reliable evidence. It must finish query understanding, filtering, retrieval, ranking, and generation within a single request, because the user is waiting.

Running both pipelines in the same synchronous interface would let a heavy document upload consume application threads, slowing online answers, and would return vague errors on parsing failures.

In FastAPI, separate routers handle different routes, services orchestrate parsing, retrieval, and chat flows, and the database module manages knowledge‑base, conversation, and task state. This layering lets errors be isolated to the appropriate layer.

Offline chain guarantees completeness and recoverability; online chain guarantees correct boundaries, low latency, and graceful degradation.

Both chains can share document IDs, knowledge‑base IDs, chunk schemas, and model configs, but they need independent task queues, resource quotas, and monitoring metrics.

Resource isolation is more than separate directories; offline parsing consumes CPU, memory, disk, and model calls, while online queries are sensitive to queueing and first‑byte latency. Sharing an unbounded thread pool can cause online requests to starve during bulk uploads.

Therefore, production design should give each chain its own concurrent entry point and queue limits, expose understandable processing status to users, and set clear timeout and degradation strategies based on real‑world load testing.

Back‑pressure is essential: when upstream uploads outpace parsing, the system must limit pending tasks per tenant, defer new uploads, or mark tasks as waiting for resources, always keeping status visible instead of silently queuing.

Resource quotas must respect tenant boundaries; a single tenant uploading large files should not drag down other tenants' online queries. Priorities, quotas, and cancellation policies are determined by load tests, but the principle "offline pressure must not monopolise online resources" must be baked in at the architecture stage.

How to split the offline ingestion chain

The ingestion pipeline can be broken into nine clear steps:

Upload validation – check identity, knowledge‑base permission, file type, size, and duplicate submissions.

Save raw file – write the original content to stable storage and create a document record and processing task.

Format routing – route PDFs, Word, PPT, plain text, and scanned pages to their respective parsers.

Structural parsing – extract text, tables, images, headings, page numbers, and positions.

Cleaning and chunking – filter useless content, preserve semantic boundaries, and retain source metadata.

Embedding computation – record model version and dimensions to avoid mixing vector spaces.

Bulk write to the search engine – assign stable IDs to each chunk and collect per‑item errors.

Integrity verification – compare expected vs. actual chunk counts, failed pages, and index status.

Publish document version – only after successful verification does the new version become visible to online queries.

Successful parsing does not equal a completed ingestion; only after write, verification, and release should the online side see the new version.

When bulk‑writing to Elasticsearch, each chunk must carry a stable business id that maps to the ES _id. After the bulk operation, errors are collected per item. Stable IDs help identify duplicate uploads, decide when to delete old version chunks, avoid half‑written versions, and ensure idempotent retries.

A robust design separates “processing version” from “current published version”. New versions are fully written and verified in the background, then atomically switched to the published pointer. If the switch fails, the old version continues serving.

The switch must preserve four invariants:

Only one clear current version is visible to online retrieval.

All chunks, vectors, and metadata belong to the same processing version.

After publishing, retrieval filters, caches, and citations must recognise the new version.

Old versions are retained only until the new version is confirmed and a rollback window expires.

Each task stage records input version, start/end timestamps, output counts, error types, and retry attempts. Deterministic errors (e.g., unsupported format) should not be retried; transient network timeouts may be retried with back‑off.

What state machine a document task needs

A realistic state machine includes more than “processing” and “completed”. Typical states are: received, awaiting parsing, parsing, awaiting embedding, writing, verifying, awaiting release, released, failed, and disabled. The state must answer where the task stopped, which artifacts exist, and where to resume.

State transitions are written by the service that completes the corresponding work; they cannot be pre‑marked as successful at request receipt.

Failures are categorised:

Retryable – temporary network errors, upstream rate‑limiting, short‑lived service outages.

Non‑retryable – unsupported format, corrupted file, insufficient permissions, schema parsing errors.

Retryable failures enter a limited‑retry queue; non‑retryable ones wait for manual intervention or user correction.

Task cancellation must clean up unpublished temporary chunks and caches without deleting the currently serving old version. Publishing should switch a single “current version” pointer atomically.

How the online query chain should work

The first step of an online request is identity and tenant scope verification, not embedding.

The backend extracts the user and tenant from the login state, determines accessible knowledge bases, documents, and versions, and injects these constraints into the retrieval filter.

Next comes query processing. Single‑turn queries go straight to retrieval; multi‑turn queries may need entity completion from history. The original question is always retained for audit.

Keyword and vector retrieval run in parallel. A typical call chain is: get_filters – convert knowledge‑base ID, document ID, and status into search conditions; get_vector – generate the vector expression; search – combine text and vector candidates; retrieval – perform re‑ranking, threshold filtering, pagination, and result assembly.

These function names are not cosmetic; they make permission checks, vectorisation, retrieval, and post‑processing testable and traceable.

Retrieval results are not the final answer. The system must assess evidence sufficiency, control context size, retain sources, and feed the question plus evidence to the LLM. The model’s streamed response returns the answer and citations.

Every online step should log its input and output under the same Trace ID, enabling root‑cause analysis when a wrong answer occurs.

Where each data type belongs

Different data have different access patterns and should not all reside in the vector store.

Relational DB – stores users, tenants, knowledge‑base metadata, document versions, task status, and permission relationships; requires constraints, transactions, and field queries.

Object storage – holds raw PDFs, PPTs, images, and parsing outputs; serves as the source for re‑parsing and reference jumps.

Search engine – stores chunk content, tokenised fields, vectors, source metadata, and filterable fields; all retrieval logic revolves around it.

Cache – keeps recomputable data such as query embeddings or limited‑scope retrieval results; it can be invalidated without breaking the system.

Log & Trace – records processing steps and diagnostics, with proper access controls for sensitive data.

Stable IDs link all storages; version fields prevent incompatibility when parsers or embedding models change.

Failure and degradation design

Failures can occur outside the LLM:

Query rewrite may select the wrong entity – fallback to the original question or ask the user for clarification.

Embedding service timeout – degrade to BM25 keyword retrieval, marking the downgrade.

Vector retrieval failure – do not let the model answer from its own knowledge; instead, return a “no evidence” response.

Rerank timeout – use the initial ranking results, recording that fine‑grained rerank was skipped.

Model service rate‑limit or outage – return retrieved source text or a clear retry prompt.

Reference verification failure – delete, downgrade, or refuse to answer high‑risk facts.

All degradation actions must be visible in the response and logs; otherwise the team only sees a successful HTTP status while the user receives a degraded answer.

Retries need boundaries: transient network glitches and rate‑limits may be retried with exponential back‑off; deterministic errors (unsupported format, permission denied) must not be retried.

Write operations must be idempotent; a client timeout that causes a duplicate request should be recognised as the same task, avoiding duplicate documents or chunks.

Common bad cases that slip through design

1. Document marked “completed” but only partially written

Partial bulk‑write failures must be detected by comparing expected vs. actual chunk counts and aggregating per‑item errors before publishing.

2. Retrieve the whole index then hide results on the front‑end

Permission filtering must happen in the retrieval stage; otherwise hidden data can still influence rerank, prompts, or logs.

3. Vector service fails and the model answers anyway

When evidence is missing, the system should either fallback to a safe retrieval path or refuse to answer, clearly marking the degradation.

4. Monitoring only interface health, not answer quality

System metrics (task backlog, latency, error rates) must be paired with quality metrics (Recall@K, no‑answer rate, citation correctness) and a bad‑case regression loop.

5. Cache returns stale content after a new version is published

Versioned keys must be part of the cache key; publishing a new version should trigger cache invalidation.

Monitoring the system and the answers

System‑level monitoring watches task queues, parsing failures, bulk‑write errors, stage latencies, model TTFT, timeouts, error rates, connection pools, and resource utilisation.

Quality monitoring watches correct evidence recall, rank of the first correct evidence, no‑answer errors, citation validity, and user feedback.

A Trace ID ties the two layers together, allowing engineers to follow a request from gateway through query processing, filtering, retrieval, rerank, prompting, model inference, and citation.

Online Trace locates errors; bad‑case regression prevents recurrence; evaluation sets verify that the next change is effective.

Acceptance criteria for a “ready‑to‑launch” version

Functional acceptance covers upload, parsing, version release, retrieval, refusal, citation, and multi‑turn dialogue.

Fault injection tests simulate parsing failures, partial writes, retrieval timeouts, rerank timeouts, model rate‑limits, and client disconnects, checking task status, degradation behaviour, and resource release.

Permission acceptance validates different users and knowledge bases, ensuring no cross‑tenant leakage in retrieval, cache, logs, or citations.

Performance acceptance measures offline throughput and online TTFT under stable concurrency, burst traffic, and cold‑start, without unsupported capacity promises.

Quality acceptance runs a fixed benchmark set, comparing Recall@K, MRR, no‑answer control, and citation correctness per question type.

Recovery acceptance verifies that interrupted tasks resume from the correct stage, old versions stay serving when new versions fail, retries are not duplicated, and no stale data remains.

All acceptance results are tied to code, model, knowledge‑base, and schema versions to ensure reproducibility.

Acceptance records must also answer “what residual state does the system leave after a failure?” – e.g., temporary chunks, duplicate tasks, stale cache entries, or orphaned references must be inspected.

Final interview answer

I would split RAG into offline ingestion and online query pipelines. Offline handles upload, format routing, parsing, cleaning, chunking, embedding, bulk indexing, integrity checks, and version release, each with task status and recoverable boundaries. Only after validation does a new version become visible; otherwise the old version continues serving.

Online starts with identity and tenant scope verification, then proceeds through query handling, permission filtering, keyword & vector retrieval, fusion, rerank, evidence thresholding, context building, and streaming generation.

Data responsibilities are separated: relational DB for users, permissions, versions, and task state; object storage for raw files; search engine for chunks, vectors, and metadata; cache for recomputable data with tenant/version keys. Failures trigger module‑level retries or graceful degradation, and insufficient evidence leads to refusal rather than a fabricated answer.

Each request carries a Trace ID linking the original query, rewrites, filters, candidates, scores, final context, model version, and citations. System monitoring tracks task health, latency, and errors; quality monitoring tracks recall, no‑answer rates, and citation correctness. Bad cases feed back into regression tests, forming a closed loop.

The bottom line is a clear system boundary, no half‑finished data leaking into online service, and the ability to locate, reproduce, and learn from any wrong answer.