Lakehouse + AI: Vector Database as Derived Index, Not New System

1. Problem Background and Technical Evolution

The most practical path for enterprise LLM adoption is RAG (Retrieval-Augmented Generation): retrieve internal documents, product knowledge, and business data as context for the LLM. These data already reside in the lakehouse — structured tables in Paimon/StarRocks, unstructured documents scattered elsewhere. Many teams build isolated "vector database silos" for RAG, causing consistency and governance gaps: business data updates but vectors stay stale; deleted documents still get retrieved; permissions managed in lakehouse are not synced to the vector store.

The core contradiction Lakehouse+AI solves is consistency and governance : let unstructured data use vector search while structured data uses existing SQL, both within the same data asset system with unified permissions and governance. This article focuses on how Milvus integrates with the lakehouse in production, how to build embedding pipelines, optimize recall, and address often-overlooked AI data governance.

2. What You Will Gain

Clear understanding of the complete RAG pipeline in lakehouse architecture: documents/business data → Embedding → Vector DB → Retrieval & Rerank → LLM.

Understanding of embedding pipeline division between offline index building and real-time updates (Flink real-time vector writes).

Mastery of Milvus vector index (HNSW/IVF) trade-offs among recall, latency, and memory .

Knowing where to tune when recall is poor : chunk splitting, embedding model, hybrid search, rerank.

Establishing consistency and AI data governance between lakehouse and vector DB (update linkage, permissions, masking, document versioning).

Ability to answer real interview questions like "Why does enterprise RAG give irrelevant answers after launch?"

3. Core Concepts and Architectural Positioning

RAG essence is " retrieve first, generate later ": user query → vectorize → retrieve top-k relevant chunks from vector DB → assemble context → LLM answers. Three storage roles:

Structured business data stays in Paimon/StarRocks, queried via SQL.

Unstructured document chunks after embedding become vector indexes, queried via vector similarity.

Original text and metadata remain in lakehouse/object storage; vector DB typically stores only embedding vectors plus doc_id, chunk_id, permission tags needed for retrieval — not the authoritative original text.

This positioning is critical : the vector DB is not a new data lake; it is a derived retrieval index/storage layer on top of the lakehouse . Authoritative document versions, permissions, lifecycle management remain with lakehouse or original business systems; the vector DB must follow authoritative data changes. Understanding this answers all subsequent consistency and governance questions.

4. Deep Dive into Underlying Principles

How Embedding Works

Embedding models map text to high-dimensional vectors (hundreds to thousands of dimensions); semantically similar texts have higher vector similarity. Retrieval encodes user query into same-dimension vector, finds top-k most similar chunks in vector DB. This enables "search by meaning" not just keywords. Limitation: embedding model capability directly affects recall — domain knowledge gaps, terminology, multilingual mixing cannot be fixed by index tuning alone.

Why Vector Indexes Trade Precision for Speed

Brute-force similarity calculation over all vectors becomes unacceptably slow at scale. Approximate Nearest Neighbor (ANN) indexes trade "missing as few targets as possible" for query efficiency. For HNSW (graph index): vectors organized in multi-layer neighbor graphs; search starts at higher layers, descends, visits limited candidate nodes. Build and search parameters affect graph connectivity, search scope, recall, and resource consumption. Larger search scope → higher recall but higher query cost; larger graph parameters → higher build cost and memory. IVF indexes cluster vectors first, search only relevant clusters; adjust probe count to trade recall vs cost. No free high recall — every recall improvement costs compute, memory, or latency.

Hybrid Search and Rerank

Pure vector semantic search misses "must exactly match ID, SKU, date, model" scenarios. Production uses hybrid search : vector search for semantic similarity, keyword search / scalar filters / structured conditions for exact constraints, fuse results. Then add a rerank stage — a higher-cost model recomputes relevance on initial candidates, selects truly relevant chunks for LLM. This "fast index coarse filter + precise model rerank" two-stage approach needs business data and SLA validation for candidate size and adoption.

5. Core Pipelines and Key Processes

Offline Index Building Pipeline (Initial Data)

Pull documents from lakehouse/object storage → clean & split into chunks → batch embedding → write to Milvus with chunk's doc_id, permission tags, business filter fields. Key is chunk splitting strategy : too large mixes topics → noisy retrieval; too small loses context → LLM can't understand. Typically split by semantic boundaries / fixed length with overlap, prepend titles and necessary metadata into chunks.

Real-time Update Pipeline (Lakehouse Integration)

When documents or business data change, vector index must add/update/delete per change strategy. This reuses the data pipelines from earlier articles — document change events flow via Kafka/Fluss into Flink, Flink re-embeds changed content, updates Milvus vectors and metadata for corresponding doc_id/chunk_id. Consistent with Articles 10-11, this pipeline can leverage Flink Checkpoint for failure recovery and state consistency boundaries, but Checkpoint alone does not guarantee end-to-end Exactly-Once for external systems like Milvus ; actual consistency depends on external write idempotency, transaction support, and retry policies. Hardest part is delete/update consistency — deleted documents must have old vectors removed or marked unsearchable by version/ID strategy, otherwise stale content gets retrieved.

6. Real Business Scenarios and Engineering Practice

Example: enterprise intelligent customer service / internal knowledge assistant. Sources: product manuals, ticket history, regulations. Steps:

Index Building

Documents land in lakehouse/object storage (authoritative source) → offline chunk, embed, load into Milvus with metadata: doc_id, dept (department), permission_tag.

Online Retrieval

User question → vectorize → Milvus semantic recall top-k, combined with user identity/authorization for filtering (unauthorized docs excluded before candidate formation) → rerank → top chunks into Prompt → LLM answers with original source links.

Key Engineering Decisions

Permission filtering at retrieval layer — don't retrieve unauthorized content then filter; avoids wasted retrieval and sensitive content leakage risk.

After embedding model is fixed , production must keep vector space and model version consistent; model upgrades usually require re-encoding historical data or explicit version migration — never mix vectors from different models, dimensions, or semantic spaces in same index.

Chunk and rerank strategies must be tuned repeatedly with real Q&A samples, not set by guesswork .

7. Performance, Cost, and Technology Selection

Milvus-type vector DB costs concentrate in three areas:

Index memory and compute resources (different index types have different resource models; HNSW graph structure adds memory overhead).

Embedding compute cost (offline batch building is fine; real-time updates invoke model per change — major GPU/remote API cost source).

Storage and data replicas (vectors, indexes, metadata all consume space).

Recall optimization should climb cost ladder: first tune chunk and retrieval parameters; if insufficient, evaluate hybrid search/rerank; last consider stronger embedding model — each step up typically means higher compute, storage, or ops cost.

Selection must respect boundaries: vector DB excels at semantic similarity search, not a replacement for analytical DBs doing complex exact-condition aggregations. "Top 10 by sales volume" still goes to StarRocks; "find products most similar to this description" suits vector search. Don't use vector DB for what it's not good at — this is core to lakehouse+AI division of labor. All recall, P99 latency, write QPS are tightly coupled to data scale, distribution, embedding model, index params, hardware, concurrency — must be benchmarked.

8. Production Issues and Troubleshooting

Issue 1: Irrelevant Answers, Poor Recall

Symptom: LLM answers unrelated to question. Follow "phenomenon → metric → principle → locate → fix": sequentially check — chunk splitting unreasonable (too large noisy / too small missing context), embedding model mismatched domain, missing permission filters, need hybrid search/rerank; also inspect actual retrieved chunks, not just final LLM answer. Fix from data splitting and retrieval params first, then consider model change.

Issue 2: Document Updated but Q&A Still Shows Old Content

Symptom: knowledge base updated, RAG still cites old version. Root cause: real-time update pipeline lagging, update failed, or old vectors not properly deleted/invalidated. Locate: check if document change events entered Flink, embedding succeeded, Milvus new/old doc_id/chunk_id and version status correct; fix: compensate failed events, delete/invalidate old vectors, re-run incremental sync. Best to add version number / update timestamp metadata so retrieval can explicitly exclude expired versions. This is RAG's most hidden and trust-damaging issue.

Issue 3: Slow Writes, Index Build Can't Keep Up with Updates

Symptom: embedding real-time writes queue up. Root cause: embedding calls (GPU/remote API) bottleneck, or Milvus write/flush/index build/resource limits bottleneck. Locate: separately measure model call latency, queue length, Milvus write throughput, flush/index build status, resource usage; fix: batch calls, cache vectors for unchanged content, control concurrency and batch size, choose real-time vs async index build per freshness requirements.

Issue 4: High Retrieval Latency

Symptom: top-k recall slows. Root cause: ANN search params too large, candidate set too large, filter conditions degrading efficiency, concurrency spike, or resource pressure from data growth. Fix: adjust index/search params per SLA benchmarks, design filter fields wisely, add resources or shards, monitor P95/P99 not just average latency.

9. Advanced Interview and Architecture Thinking

Q1: Why does enterprise RAG give irrelevant answers after launch?

Key: problem often in data and retrieval pipeline, not generation model — chunk splitting, embedding match, missing hybrid search/rerank, recall quality, permission filters can all cause answer drift; check actual retrieved content first, then decide if generation model needs adjustment.

Q2: How to keep vector DB consistent when documents update?

Key: vectors are derived indexes of documents, must add/update/delete with document changes; rely on Flink pipeline + doc_id/chunk_id linkage + version control + delete/invalidate old vectors; Checkpoint provides failure recovery but cannot equal end-to-end Exactly-Once for external vector DB — still need idempotency, compensation, consistency verification.

Q3: How to trade off HNSW parameters for recall vs latency?

Key: graph build params and search width affect index memory, build cost, query latency, recall; larger search scope usually higher recall but higher query cost; specific values must be pressure-tested against business SLA and actual dataset.

Q4: How to unify permissions between vector DB and lakehouse original system?

Key: permission info as retrieval metadata or filter conditions participates in candidate generation; filter at retrieval layer as early as possible; but cannot assume "vector DB filtered once is absolutely safe" — final data access must still combine business system auth model and data security policies for verification.

Q5 (Connecting Question): Why is lakehouse+AI not building a new system?

Key: authoritative data, versions, permissions remain in lakehouse or original business systems; vector DB is just derived retrieval index; previous CDC/Flink/Checkpoint pipelines reused as real-time vector update pipeline; versioning, idempotency, compensation mechanisms solve derived index consistency.

10. Summary, Knowledge Extraction, and Next Episode

This article connected lakehouse to LLM: RAG = data → embedding → vector DB → retrieval & rerank → LLM generation; offline build and real-time update pipelines both reuse Flink data pipelines; Milvus index balances recall, latency, memory, compute cost; poor recall debug from chunk, embedding, hybrid search, rerank; AI data governance key is vectors follow document changes, version control, permission pushdown, authoritative source stays in lakehouse or original business system. Per Chapter 2: we clarified full pipeline, embedding pipeline, index trade-offs, recall optimization, consistency governance.

One-sentence conclusion: Enterprise RAG success depends not only on the LLM itself, but on whether the retrieval results fed to it are accurate, fresh, and permissioned — the vector DB is a derived retrieval index of the lakehouse or business data, not the data authority.

Next episode (Article 16, Season 2 finale): Lakehouse Platform Architecture Evolution and Technical Debt Governance — 0-to-1 roadmap, 1-to-N multi-cluster multi-tenant, storage-compute separation migration, technical debt governance, team and cost governance, plus full Season 2 summary.