How Milvus 3.0 Queries External Lakehouse Files Without Copying Data

1. External Collection is a Query View, Not a Writable Table

Translating External Collection as an "external table" can mislead; the user still interacts with a Collection via the usual create, load, search, and query APIs. Only refresh‑related interfaces are added. The query side continues to use existing schema, segment, load, index, and search concepts, while the read‑only boundary is enforced in the Proxy by blocking insert, delete, upsert, and flush operations on fields marked with external_field. Import, manual compaction, and schema modifications such as user primary key, dynamic fields, partition key, clustering key, or auto ID are unsupported.

2. Tightening the Boundary with external_source and external_spec

External Collection must answer where the data lives ( external_source) and how to read it ( external_spec). The source URI is allow‑listed (s3, s3a, aws, minio, oss, cos, obs, gs, gcs, azure) and must include a host; userinfo or clear credentials are rejected. Validation occurs in ValidateSourceAndSpec (Proxy and RootCoord) and ValidateExtfsComplete, which requires an explicit cloud_provider and a single authentication mode (AK/SK, role ARN, IAM, GCP impersonation, or anonymous). Redaction of sensitive fields is performed by RedactExternalSpec.

3. Stable File Lists: The First Pitfall Is Not the Network

Refresh does not simply scan a bucket; it first explores files via ExploreFilesReturnManifestPath, writes an explore manifest, then calls NormalizeFileInfos. Files are sorted lexicographically by FilePath and filtered by extension (parquet, vortex). Arrow filesystem's GetFileInfo does not guarantee order, so Milvus enforces deterministic ordering to avoid mismatched index windows that could cause silent data loss or Invalid parquet magic errors.

4. Refresh Recalculates Segment Mapping Instead of Re‑importing

Refresh is a manual, sub‑second operation described in the user docs, but the article notes the lack of independent benchmarks. Internally, a Refresh creates a Job and multiple Tasks. The Manager validates collection state, rejects concurrent jobs, and persists the Job in Init state. After I/O, the explore phase generates a shared manifest, which is sliced into file windows for Tasks. The Job state machine uses index.JobState values (Init, InProgress, Finished, Failed). DataNode processes each window, reading fragments identified by FilePath + StartRow + EndRow. Refresh compares existing fragments to new ones, keeping unchanged segments, patching those with missing fields, and creating new segments for orphan fragments. The process respects schema evolution rules: additive changes can self‑heal on the next refresh, while drops, renames, or type changes require stronger coordination.

5. Visibility Depends on Atomic Segment Replacement

Task completion does not guarantee query visibility. Only after the Job reaches Finished does the Manager call applyJobInfo, atomically updating all kept and updated segments in a critical section. The checker ensures kept segments exist, updated segments have manifest and fake binlog, and row counts match. QueryNode then loads the new manifest, ensuring queries see a consistent view.

6. Virtual Primary Key Enables External Rows in the Existing Query Model

When the source lacks a primary key, Proxy injects __virtual_pk__ computed as

((segment_id & 0xffffffff) << 32) | (row_offset & 0xffffffff)

. The high 32 bits come from the segment ID, low 32 bits from the row offset. QueryNode can infer the originating segment from the shifted value. Collision detection is performed by detectVirtualPKCollisions, and the system treats collisions as a monitorable constraint rather than a guarantee of uniqueness.

7. milvus‑table: Connecting Snapshots to the External Query Chain

External Collection also supports milvus-table format, allowing Milvus’s own snapshots to serve as an external source. When external_spec.format=milvus-table, external_source points to a snapshot metadata JSON. RootCoord reads the snapshot schema, validates identity, and maps external_field to source field IDs. DataCoord requires the metadata to contain storagev2_manifest_list. The milvus_table.go module converts the source manifest to FileInfo and attaches L0 delete overlays. DataNode creates a one‑to‑one fragment‑to‑segment mapping for milvus-table, preserving the relationship between source segment and row offset.

Summary

External Collection’s significance lies in turning external lakehouse files into a Milvus‑managed query view. The external_source and external_spec definitions tighten access boundaries; DataCoord builds a deterministic file view; Refresh recomputes segment mappings via Job/Task and fragment identity; virtual PKs and milvus-table snapshots integrate files without native primary keys into the existing query model, while preserving Milvus‑managed indexes and derived state. The implementation shows Milvus extending its query boundary from internal data copies to external data views, but full zero‑copy lakehouse querying still requires further validation and benchmarking.