Milvus 3.x: From Vector Database to Lake-Native AI Retrieval Engine

Introduction: Three Generations of Retrieval

The article frames data retrieval evolution in three generations: (1) relational databases with exact SQL matching, (2) keyword search engines like Elasticsearch using inverted indexes and BM25, and (3) vector search that maps data to embeddings for semantic similarity via cosine, inner product, or L2 distance. It emphasizes that vector dimensions are model-dependent, not fixed to 768/1024/1536.

RAG (Retrieval-Augmented Generation) accelerates vector search adoption. Milvus 3.0, released July 29, 2026, marks a pivotal shift: Milvus is no longer just a vector database but a Lake-Native Vector Search / Retrieval Engine . Core 3.x features include External Collection, Storage V3, Snapshot, online schema evolution, SINDI sparse index, StructArray, faceted search, TEXT type, and FAISS Passthrough.

Core Architecture

Milvus retains a decoupled compute-storage, control-data plane separation design with four layers:

Proxy : stateless gateway for request routing and aggregation.

Coordinator : cluster topology, DDL/DCL, timestamps, streaming service, segment/compaction/index coordination.

Streaming Node : handles WAL, growing data, shards for real-time ingestion.

Query Node : serves historical data queries.

Data Node : processes historical data, compaction, index building.

The architecture maintains streaming vs. historical query separation. The key 3.x advance: data need not be copied into Milvus-managed storage .

Lake-Native Architecture & External Collection

Traditional RAG pipelines duplicate data: raw documents in a data lake (Parquet, Iceberg) and vectors in Milvus, causing redundancy, complex sync, inconsistency, and fragmented governance.

External Collection provides a third path: Milvus builds vector indexes and metadata directly on lake-native files (Parquet, Iceberg, Lance, Vortex) without copying source data. Benefits:

Eliminates data duplication — only indexes stored in Milvus.

Single source of truth — avoids version drift.

Simplifies sync — only index updates needed.

Unified governance — permissions, lineage, lifecycle stay in lake.

Caveats : "Zero-copy" applies to source data only; index storage, manifest metadata, query cache, compute, and object storage I/O still incur costs. External Collection is read-only and not a real-time CDC system; lake changes require manual Refresh to update manifests and indexes.

Storage V3 & Snapshot

Storage V3 introduces versioned storage layout with a Manifest-based organization, enabling continuous data evolution, Snapshot, and External Collection. It supports Parquet, Vortex, etc. It is not merely a file format change; the combination

Versioned Storage + Manifest + Object Storage + Snapshot + External Collection

forms the Lake-Native data foundation.

Snapshot provides point-in-time views of a collection while online serving continues, enabling stable datasets for offline processing, embedding rebuilds, evaluation, and data versioning.

Vector Similarity Metrics

Three metrics: Cosine (angle, ignores magnitude, common for normalized embeddings), Inner Product (direction + magnitude), L2 (geometric distance). No absolute rule like "text → Cosine, image → L2, recommendation → IP". The engineering principle: distance metric must align with embedding model training objective, normalization, and offline evaluation results .

ANN Indexes: HNSW, IVF, DiskANN

HNSW

Graph-based index. Parameters: M (graph connectivity), efConstruction (build-time candidate pool), ef (query-time search scope). Higher ef improves recall but increases latency; actual trade-offs require business dataset validation.

IVF

Cluster-then-search. Parameters: nlist (number of clusters), nprobe (clusters probed at query time). Larger nprobe expands search but gains must be verified via benchmarking.

DiskANN

Disk-aware index for datasets exceeding memory. Leverages NVMe SSDs. Selection logic: assess if data fits in memory → need high recall → accept disk access → SSD performance → benchmark decision. Official benchmarks show compressed BM25 index ~1/3 size of Milvus 2.6 sparse index at similar recall; SINDI achieves up to 10x QPS over MaxScore on learned sparse embeddings (specific benchmark, not universal guarantee).

Multi-Vector & StructArray

Real documents contain titles, paragraphs, images, tables. Traditional "1 entity = 1 vector" flat model limits granularity. Milvus 3.x introduces StructArray and embedding lists, allowing a document to be split into multiple chunks each with its own embedding, enabling element-level retrieval for complex documents, multimodal RAG, and long-document fine-grained search.

Data Ingestion Paths

Traditional Managed Collection

Data flows: write → streaming path (WAL) → growing segments → compaction/index build → historical segments served by Query Nodes.

External Collection Path

Source data remains in lake (Paimon, Iceberg, Parquet). Milvus builds indexes directly on lake files via

Lake → External Collection → Index Build → Retrieval Service

. Avoids full copy, but requires manual refresh on source changes.

Query Pipeline

End-to-end intelligent search:

User Query → Query Rewrite → Embedding → Vector/Sparse Search → Filter → Hybrid Search → Rerank → Context Assembly → LLM

. Milvus 3.x pushes post-processing (sorting, aggregation, faceted search, grouping) into the engine, evolving from pure Top-K vector search to a full retrieval processing engine .

Production Scenarios

Enterprise Knowledge Base RAG (Paimon + Milvus + LLM)

Raw docs (PDF, Word, Wiki) land in object storage/Paimon/Iceberg. Spark/Flink or doc service handles: Parse → Clean → Chunk → Embed → Milvus Upsert. Chunking should follow semantic boundaries (headings, paragraphs) not fixed token counts; evaluate chunk size/overlap offline. Embedding model selection (BGE-M3, commercial APIs, custom) must consider language coverage, dimension, context length, domain, cost.

Collection schema example:

chunk_id (PK), embedding (vector), document_id, tenant_id, business_domain, permission, create_time, source_uri (scalars)

. Index: start with AUTOINDEX; evaluate HNSW for hot high-recall data.

Query flow:

Question → Rewrite → Embed → Milvus Top-N → Permission Filter → Rerank → Top-K → Context → LLM → Answer + Citation

. Permission filtering at retrieval stage.

Best practices: semantic chunking; matching doc/query embedding models; two-stage retrieval (vector Top-N → rerank → Top-K) with N/K tuned via evaluation set; upsert with stable PK and version/timestamp for incremental updates; permission as filter predicate.

E-commerce Real-Time Recommendation (Flink + Milvus)

MySQL binlog → Flink CDC → Kafka → Flink cleaning/feature engineering → Embedding service → Milvus Upsert. Corrects a misconception: "hot items have larger vector magnitude → use IP" is invalid. Metric choice (Cosine/IP/L2) must derive from model training objective and offline Recall/NDCG. If embeddings are normalized, IP and Cosine are rank-equivalent.

Query:

Item Vector → Milvus Top-N → Inventory/Status Filter → User Feature Fusion → Rerank → Top-K

. High concurrency via Query Node scaling, replicas, batch queries, GPU acceleration; but no fixed SLA (e.g., 50ms) without hardware+data benchmark.

Technology Selection

Comparison matrix (image referenced). Guidance:

Milvus : core need is professional vector search at scale, horizontal scaling, multiple ANN indexes, dedicated vector infrastructure.

Elasticsearch : already deep in ES, primary need full-text/keyword search with added vector capability.

pgvector : limited scale/concurrency, existing PostgreSQL stack, want SQL-unified scalar+vector queries.

Weaviate : rapid AI app building with modular vector search + AI workflows.

Reject hard thresholds (e.g., "Milvus >10M vectors"). Real selection based on: data scale + QPS + P99 + Recall@K + update frequency + query complexity + existing stack + ops capability + cost .

Performance Tuning

HNSW

Focus on M, efConstruction, ef. ef is primary online recall/latency knob. Avoid fixed rules like "ef=64 → 95% recall". Build 3D curve: Recall@K ↔ P95/P99 ↔ QPS using business evaluation set.

IVF

Tune nlist, nprobe. Larger nprobe widens search; validate gains via load test.

DiskANN

Requires high-performance NVMe SSD. Monitor SSD perf, data scale, memory, search params, Query Node local disk config.

GPU_CAGRA

Watch GPU memory, batch size, query pressure, GPU utilization. GPU not universally faster; benefit tied to request pressure and batch density.

Sharding & Caching

Shard count not simply "Query Nodes × 2/4". Plan for write throughput, query parallelism, data distribution, future scaling. Over-sharding increases merge/management overhead; under-sharding limits parallelism. Cache hit rate >90% does not guarantee millisecond latency; actual latency depends on index type, segment count, data scale, Query Node CPU/memory, object storage, network, Top-K, filters.

Cost Model

Traditional: Lake + Milvus Storage + Index + Compute (potentially two data copies). Lake-Native External Collection: Lake + Index + Compute (reduces replication). But avoid claiming "50% storage cost reduction". Actual cost drivers:

Index, Cache, Compute, Object Storage, Network, Refresh, Data Volume

. Core value: eliminate unnecessary copies, make lake the source of truth, bring vector search to data .

Production Troubleshooting

Low Recall

Investigate upstream to downstream: Ground Truth label quality → embedding model quality/consistency → chunking strategy/granularity → similarity metric choice → filter scope → index type → search params → reranker strategy. Do not default to increasing ef/nprobe ; root cause often in data layer (vector quality, chunking, metric mismatch, over-filtering). Index params are last-step optimization.

High Query Latency

Check: QPS level, P99 distribution, Query Node CPU/memory, segment count/state, index type, filter complexity, Top-K, object storage I/O, cache hit rate. For External Collection add: external source access latency, object storage bandwidth, manifest size/load speed, index storage location/read cost, refresh frequency/duration. Lake-Native does not eliminate object storage cost; latency heavily depends on underlying storage path.

Query Node OOM

Dimensions: raw vector volume, index file size, segment count/replicas, batch query concurrency, Top-K size, query cache usage. Memory estimate: FP32 raw vectors = count × dim × 4 bytes (e.g., 100M × 768 ≈ 307 GB). Actual runtime memory adds index, metadata, segment cache, replicas, runtime overhead — far exceeding raw vector size.

Slow Index Build

Full chain: Data Node compute load, object storage write bandwidth, segment merge strategy, compaction resource contention, index build resource quota, SSD I/O, GPU acceleration enablement, External Collection refresh end-to-end time. For External Collection, isolate bottleneck: source read slow (storage bandwidth, file scan efficiency), index build slow (compute, algorithm), search slow (query path) — not just "Query Node performance".

External Collection Data Inconsistency

Correct mental model: not real-time sync. Flow:

External Source → Refresh → Manifest Mapping → Index Update → Query Visible

. Design: define source update frequency, set matching refresh interval, confirm index build completion, verify query-visible data version. If sub-second freshness required, use Managed Collection or real-time write path; External Collection cannot replace CDC-level real-time serving.

Advanced Interview & Architectural Thinking

Q1: Biggest architectural change in Milvus 3.x?

Not just "higher performance". Evolution from Vector Database to Lake-Native Retrieval Engine . Enablers: External Collection, Storage V3, Snapshot, sparse index enhancements, StructArray/multi-vector, server-side sorting/aggregation. Most transformative: data stays in lake, Milvus builds indexes and serves search directly on lake-resident data.

Q2: What problem does External Collection solve?

Traditional "Lake → ETL → Milvus" full copy causes redundancy, complex sync, consistency issues, high storage cost, governance fragmentation. External Collection enables "Lake → External Collection → Index → Search" with source data unmoved. Boundaries: still has index/compute/storage costs, requires refresh, read-only, not real-time sync.

Q3: What is Storage V3?

Three levels: (1) new storage model replacing legacy; (2) versioned data layout with Manifest managing object storage files, columnar organization, flexible schema evolution; (3) foundational layer for Snapshot, External Collection, lake-native access. Core chain:

Storage V3 → Manifest → Snapshot → External Collection → Lake-Native Retrieval

.

Q4: Why can't RAG rely only on vector search?

Because semantic similarity ≠ business relevance . Full RAG pipeline: Query → Rewrite → Dense Vector Search → Sparse/BM25 Search → Fusion → Filter → Rerank → LLM Generation. Each component irreplaceable: dense for semantic recall, sparse/BM25 for keyword/entity exact match, filter for hard constraints (permissions, domain, time), reranker for precision ranking, LLM for reasoning/generation.

Q5: Design "Lakehouse + Milvus 3.x + RAG" production architecture.

Three-layer separation (architecture diagram referenced):

Data Lake Layer : object storage base, Paimon/Iceberg tables, stores facts/history, unified governance/batch processing. Two paths: real-time data → Milvus Managed Collection; historical lake data → External Collection.

Milvus Retrieval Layer : unified search engine. Managed Collection for real-time writes; External Collection for lake-native data. Built-in dense/sparse/hybrid search, index management, result processing. Exposes standard search APIs.

LLM & Application Layer : app does query rewrite, calls Milvus for search+filter, reranks, passes context to LLM for reasoning/answer. Clear separation of data sovereignty, retrieval capability, generation capability.

Conclusion & Knowledge Distillation

Milvus 3.x is at a key inflection point. Core value shifts from "how to search vectors faster" to "how to give open lake data direct high-performance AI retrieval capability". This forms the true Lakehouse + Vector Retrieval + AI architecture direction: data stays in place, capabilities move to data .

Next article preview: governance & operations layer (metadata, catalog, lineage, schema, quality, security, masking, audit, resource governance, cost optimization, observability, SLA) to turn component stack (Hadoop + Kafka/Fluss + Paimon/Iceberg + Flink/Spark + StarRocks + Milvus) into a governable, observable, operable, sustainable enterprise lakehouse platform.