Turning Data into Model-Ready Knowledge with RAG Pipelines and Vector DBs

Opening Scenario

A fintech customer‑service agent cited an outdated product‑rate document in its first week, causing tens of thousands of dollars in losses. The vector store lacked incremental updates, so changes in the CMS were not reflected in embeddings. In the second week the agent answered a client of Company B with contract terms from Company A because ACL information from the document layer was not propagated to the retrieval layer; the vector store stored only chunk content and doc_id, leaving tenant isolation to the application layer.

These failures are engineering flaws in the L2 data‑and‑knowledge layer, not model‑capacity limits. Stale or unauthorized knowledge leads to hallucinations or accidents.

What the L2 Layer Solves

The core problem is converting scattered enterprise data (CMS, Wiki, Confluence, S3, databases) into model‑usable knowledge. The pipeline consists of parsing, cleaning, chunking, embedding, indexing, retrieval, reranking, and permission filtering. Each stage’s technology choice directly impacts answer quality.

Retrieval, not the model, is the ceiling of RAG. Industry consensus in 2026 states: If the correct knowledge cannot be retrieved, even the strongest model will hallucinate. Teams such as Unstructured and LlamaIndex report that most apparent hallucinations stem from upstream issues: lost tables during parsing, semantic breaks during chunking, mismatched embedding models, or pure‑vector retrieval missing keyword hits.

Enterprise‑grade requirements:

Timeliness : document changes must propagate to the vector store within minutes.

Security : ACLs must be enforced at retrieval time.

Explainability : every answer must be traceable to a specific chunk and document version.

Core Architecture Overview

Data Sources : CMS, Wiki, Confluence, PDF, databases, APIs – mixed structured and unstructured.

Parsing / Cleaning : PDF table OCR, HTML boilerplate removal, Markdown structure preservation. 2026 mainstream tools include Unstructured, Marker, and PyMuPDF, which retain element types (Title, Table, ListItem, NarrativeText).

Chunking : Recursive splitting is the default; semantic splitting can boost recall by ~9 % on topic‑clear documents; contextual chunking (Anthropic style) prefixes each chunk with a document summary to further improve relevance.

Embedding : BGE‑M3 (BAAI) outputs dense, sparse, and multi‑vector representations and is cost‑effective for multilingual scenarios.

Vector Indexing : HNSW is the default; DiskANN is used for ultra‑large single‑node cases.

Hybrid Retrieval : Combine vector recall, BM25 full‑text, and knowledge‑graph traversal; fuse three result streams with Reciprocal Rank Fusion (RRF, k=60).

Rerank : Cross‑Encoder refines the top‑20 to top‑5, raising accuracy.

Permission Filtering : ACL metadata filtering must happen during retrieval, not after generation.

Golden rule: Recall wide (multi‑route), rerank hard (Cross‑Encoder), filter early (pre‑filter ACL).

Key Technology Deep‑Dive

Document Parsing & Chunking

Parsing is the most underestimated step – “garbage in, hallucination out” is the first law of RAG engineering in 2026. Using pdfplumber.extract_text() on a complex table‑rich PDF loses row‑column relationships, leaving the model with a flat string of numbers. Tools like Unstructured and Marker preserve element types, enabling structure‑aware chunking instead of naïve character‑based splits.

Chunking strategies and their impact:

Recursive Splitting : LangChain/LlamaIndex default; split paragraph → sentence → word; cheap, deterministic, language‑agnostic.

Semantic Splitting : Embed each sentence; cut when similarity drops below a threshold. Gains ~9 % recall on clearly themed documents but incurs higher indexing cost.

Late Chunking : Encode the whole document with long‑context models (BGE‑M3, Jina‑v3) then split token‑level vectors; each chunk’s embedding sees the entire document, offering the strongest context awareness.

Contextual Chunking : Anthropic style; prepend 1‑2 summary sentences to each chunk before embedding, significantly boosting recall at the cost of an LLM call per chunk during indexing.

Practical values: chunk size 400‑800 tokens, overlap ≈10 % (80‑120 tokens). Start with recursive splitting; upgrade only if recall is insufficient.

Embedding Model Selection

BGE‑M3 (BAAI) is the 2026 benchmark for Chinese/multilingual scenarios. Its three‑fold capabilities let a single model serve three purposes:

Multi‑Functionality : simultaneously outputs dense, sparse (learned BM25‑like weights), and multi‑vector (ColBERT‑style late interaction) representations.

Multi‑Linguality : supports 100+ languages.

Multi‑Granularity : up to 8192 tokens, covering short sentences to long documents.

Hybrid retrieval with BGE‑M3 produces dense and sparse vectors in one pass, eliminating a separate BM25 service. Fusion score: s = w1·s_dense + w2·s_lex + w3·s_mul Weights are tuned per downstream scenario.

Hybrid Retrieval + Rerank + Permission Filtering

Pure vector retrieval misses exact matches (e.g., product SKUs, legal clause numbers); pure BM25 misses semantic rewrites. The 2026 WANDS benchmark shows pure BM25 NDCG 0.6983, pure KNN 0.6953 (statistically indistinguishable), while hybrid reaches 0.7497 – a **7.4 % improvement**. Fusion uses RRF, which only looks at rank positions, avoiding score incompatibility.

def hybrid_retrieve(query, top_k=20, final_k=5):
    vec_results = vector_db.search(embed(query), top_k)
    bm25_results = bm25.search(query, top_k)
    kg_results = knowledge_graph.traverse(query, depth=2)
    merged = rrf_fuse([vec_results, bm25_results, kg_results])
    reranked = reranker.rerank(query, merged[:top_k], top_n=final_k)
    return [r for r in reranked if r.acl.contains(current_user.permissions)]

Key decisions encoded in the snippet:

Three‑way recall (vector + BM25 + graph) to cover diverse query types.

RRF fusion for rank‑based robustness and zero‑parameter tuning.

Early ACL pre‑filter: ACL metadata stored with each vector ( r.acl) and filtered during retrieval, preventing unauthorized chunks from ever being recalled.

Reranking uses a Cross‑Encoder (BGE‑Reranker‑v2‑m3 is lightweight; Gemma version offers higher precision). It compresses the top‑20 to top‑5, improving NDCG@3 by ~12 % while adding 100‑200 ms latency; therefore keep the candidate set small.

Incremental Index Updates

The opening “outdated document” bug stemmed from missing incremental updates. Production pipelines need CDC listeners plus a dual‑channel index update:

def sync_document_update(doc_id, change_type):
    if change_type == "modified":
        vector_db.delete(filter={"doc_id": doc_id})
        chunks = chunker.split(parser.parse(doc_id))
        embeddings = embed_model.batch_encode(chunks)
        vector_db.upsert(doc_id, chunks, embeddings, version=increment(doc_id))
    elif change_type == "deleted":
        vector_db.delete(filter={"doc_id": doc_id})

Key points:

Delete‑then‑insert for modifications to avoid duplicate versions.

Version tag on each chunk enables retrieval of the latest version or multi‑version comparison.

Explicit delete for deletions; otherwise “ghost documents” remain searchable.

CDC from the upstream CMS triggers this function, targeting <5 minute latency.

Open‑Source Vector DB Comparison (2026)

Milvus 3.0‑beta – Self‑hosted/K8s – p99 latency ~30 ms – Recall 96 % – Suitable for billions‑scale vectors, data‑lake use case – Features: distributed, zero‑copy Parquet/Iceberg, native BM25.

Qdrant 1.18 – Self‑hosted/Cloud – p99 latency 24 ms – Recall 96 % – Latency‑sensitive large scale – Features: Rust implementation, TurboQuant quantization, Filterable HNSW.

Weaviate 1.37 – Self‑hosted/Cloud – p99 latency 38 ms – Recall 95 % – Multimodal RAG, multi‑tenant – Features: GraphQL API, native hybrid BM25, MCP Server.

pgvector 0.9 – Postgres plugin – p99 latency 62 ms (95 ms filtered) – Recall 97 % – Existing Postgres stacks, <100 M vectors – Features: iterative scan, zero ops, CVE‑2026‑3172 patched.

Chroma 1.5 – Embedded/Cluster – p99 latency 110 ms – Recall 94 % – Prototyping, local‑first – Features: DuckDB backend, best developer experience.

Pinecone Serverless – Fully managed SaaS – p99 latency 140 ms – Recall 95 % – Zero ops, 100 M+ vectors – Features: $20/mo Builder tier, cross‑region latency variance.

2026 highlights:

pgvector 0.9’s iterative scan makes filtered HNSW practical, removing the need for a 10× ef_search boost.

Qdrant 1.12’s GPU index build reduces 100 M vector initial build from 8 h to 35 min; 1.18’s TurboQuant improves recall at equal compression.

Milvus 3.0‑beta’s zero‑copy enables direct Parquet/Iceberg queries, turning the vector store into a data‑lake compute engine.

pgvector 0.8.2 patched CVE‑2026‑3172 (buffer overflow).

Weaviate v1.37 added MCP Server, standardizing the Model Context Protocol for Agent‑to‑vector‑store calls.

Selection guidance:

Enterprise‑scale (>1 B vectors) → Milvus.

Latency‑critical workloads → Qdrant.

Existing Postgres <100 M vectors → pgvector.

Multimodal or multi‑tenant scenarios → Weaviate.

Prototyping → Chroma.

Fully managed with budget → Pinecone.

Production Implementation Blueprint

Full‑plus‑Incremental Dual Pipelines

Two parallel indexing pipelines are required:

Full‑load pipeline : Rebuild the entire index weekly or monthly to correct drift, switch embedding models, or adjust chunk strategies.

Incremental pipeline : Driven by CDC events, applies changes within minutes. Each pipeline writes to a separate collection (versioned). Traffic can be gray‑scaled between them to isolate bugs.

Embedding model upgrades invalidate old vectors. Tag each collection with embedding_model_version, load the new model into a fresh collection, run shadow traffic to compare recall, and switch only after no regression.

RAG Quality Tuning Experiments

Three most impactful knobs:

Chunk size (400 vs 600 vs 800 tokens).

Top‑K recall count (20 vs 50).

Rerank Top‑N (3 vs 5).

Experiment loop (using RAGAS and a Golden Set):

def rag_tuning_experiment(configs, golden_set):
    results = {}
    for cfg in configs:
        index = rebuild_index(chunk_size=cfg.chunk_size, overlap=cfg.chunk_size // 10)
        for query, expected in golden_set:
            ctx = hybrid_retrieve(query, top_k=cfg.top_k, final_k=cfg.final_k)
            answer = llm.generate(query, ctx)
            score = ragas.evaluate(answer, ctx, expected)
            results.setdefault(cfg, []).append(score)
    return best_config(results, metric="faithfulness")

Record chunk_size, top_k, final_k together with Faithfulness and Context Recall; pick the Pareto‑optimal configuration. Keep LLM and embedding versions fixed during retrieval‑parameter sweeps.

Agentic RAG: Letting the Agent Drive Retrieval

Standard RAG follows a static “query → retrieve → answer” flow, which fails on multi‑step questions. Agentic RAG gives the LLM control over retrieval loops:

def agentic_rag(agent, query, max_rounds=3):
    context = ""
    for _ in range(max_rounds):
        action = agent.decide(query, context)
        if action.type == "search":
            rewritten = agent.rewrite_query(query, context)
            results = hybrid_retrieve(rewritten)
            context += results
        elif action.type == "answer":
            return agent.generate(query, context)
        elif action.type == "ask_user":
            return action.clarification
    return agent.generate(query, context)

Key actions: search: rewrite query, retrieve, accumulate results. answer: generate when context is sufficient. ask_user: request clarification when information is missing. max_rounds caps the loop to avoid infinite retrieval; the opening incident of an agent calling the same tool 47 times lacked such a limit.

Agentic RAG adds 3‑10× latency and linear cost per round. 2026 consensus recommends perfecting Advanced RAG (hybrid + rerank + evaluation) before adding Agentic loops. A pragmatic approach mixes both: default to Advanced RAG, switch to Agentic only for detected multi‑hop queries.

MCP Standardization

The Model Context Protocol (MCP) became the 2026 standard interface between RAG layers and agents. Weaviate v1.37’s MCP Server lets agents retrieve knowledge via a uniform API, eliminating custom adapters per vector store. Combining MCP with Agent‑to‑Agent (A2A) architectures reportedly speeds workflow development by 40‑60 %.

Metrics and Acceptance Criteria

Retrieval Faithfulness ≥ 85 % (RAGAS).

Context Recall ≥ 80 % (Golden Set).

Rerank latency < 200 ms (end‑to‑end).

Incremental update latency < 5 min (document‑to‑index).

Permission‑filter coverage = 100 % (audit logs).

Retrieval P99 latency < 500 ms (trace stats).

Hybrid vs. pure vector accuracy improvement ≥ 10 % (A/B test).

Red lines during acceptance:

100 % ACL coverage – validated with adversarial queries.

Incremental delay < 5 min – otherwise users receive stale knowledge.

Faithfulness ≥ 85 % – below this hallucinations become noticeable.

Common Pitfalls & Best Practices

Chunking too fine breaks semantics. Use recursive split + ~10 % overlap; keep tables/code blocks whole; upgrade to semantic or contextual chunking only after baseline recall is insufficient.

Ignoring ACLs leads to data leaks. Store ACL metadata in the vector store and pre‑filter at retrieval; test with adversarial queries.

Rerank latency too high . Limit vector recall to Top‑20, rerank to Top‑5; use lightweight Cross‑Encoder (e.g., BGE‑Reranker‑v2‑minicpm‑layerwise) with layer‑wise truncation.

Pure vector retrieval misses keywords . Combine vector recall with BM25; BGE‑M3’s simultaneous dense + sparse output removes the need for a separate BM25 service.

Missing delete events in incremental updates . CDC must capture insert/update/delete; deleted events must truly remove vectors to avoid “ghost documents”.

Changing embedding model without full re‑index . Delete old vectors and upsert with the new model; version collections and use shadow traffic for safe rollout.

Rerank shuffles chunk order . Place the most relevant chunks at the beginning and end of the context window where LLM attention is strongest.

Using “latest” Docker tags . Pin dependencies to exact SemVer; 2026 data shows “latest” caused 78 % of rollbacks due to hidden CVE patches or quantization changes.

Relationship with Upstream/Downstream Layers

Upstream L0 provides storage and compute (SSD IOPS affect HNSW load time; GPU resources power embedding services). Downstream L3 (Prompt & Context layer) consumes the chunks retrieved by L2; retrieval quality directly determines L3 context quality and thus L1 model output. L4 agents invoke L2 repeatedly (Agentic RAG), and L6 long‑term memory stores reusable vectors. MCP standardizes the L2‑L4 interface.

Summary

Problem solved : Transform enterprise knowledge into model‑usable context.

Core tech : Document parsing, chunking, hybrid retrieval, rerank, permission inheritance.

Preferred stack : Milvus or Qdrant (vector DB), BGE‑M3 (embedding), BGE‑Reranker‑v2 (rerank).

Key metrics : Faithfulness ≥ 85 %, Recall ≥ 80 %, Rerank < 200 ms, Incremental < 5 min.

2026 trends : pgvector 0.9 iterative scan, Qdrant TurboQuant, Milvus 3.0 data‑lake, MCP standardization, Agentic RAG.

Biggest pitfalls : Over‑fragmented chunking, ACL leaks, vector‑only retrieval, forgetting full re‑index on model change.

One‑liner : If the correct knowledge cannot be retrieved, even the strongest model will hallucinate.