Milvus 3.0 External Tables: Patch‑Add Columns and Query Snapshots as Tables

1. Function‑output fields: extending a column on top of the external source

Milvus validates schema fields in two mutually exclusive classes:

External input field : external_field is required, source column must exist, manifest column name is the literal external_field value.

Function‑output field : external_field is forbidden, no source column check, manifest column name is a decimal fieldID string.

Function‑output fields such as BM25 sparse vectors, MinHash signatures, or text embeddings cannot be declared with external_field because they have no counterpart in the source table; they must be declared using a decimal fieldID string. Only one external_field may map to a user field; any conflict triggers an error.

The Proxy creates a collection by first validating function‑output fields, then performing external schema validation. Reversing this order causes the function‑output field to be treated as an unmapped external field and rejected, as noted in the design document.

Dual‑track column naming is a subtle pitfall: external source columns use the literal external_field value to match the Parquet schema, while function‑output columns use the decimal fieldID string to match internal StorageV3 conventions. C++ parsing first looks up external_field; if not found it falls back to the numeric field ID.

During a Refresh, the function runs as a streaming pipeline: an input manifest containing only external columns is read, Arrow batches are streamed, the function processes each batch, and the output columns are written into a new StorageV3 packed column group. Peak memory equals one Arrow batch (default 64 MiB) and is independent of segment size.

The execution order of function‑output fields matters: TextEmbedding → BM25 → MinHash, to avoid planner interference from a pre‑filled BM25 column.

After the function finishes, the output fields become ordinary schema fields. Indexing follows the manifest’s field IDs: BM25 sparse uses a sparse inverted index, MinHash binary uses MINHASH_LSH, and TextEmbedding float uses a regular vector index.

Stats visibility is handled separately: during Refresh DataNode aggregates BM25 outputs into a stats blob keyed by bm25.<fieldID>, writes the stats, then commits the manifest. The manifest acts as the visibility commit point; retries overwrite the same deterministic path, and readers see stats only after the manifest commit.

Text index stats are built asynchronously: Refresh does not create the text index; DataCoord triggers an external task, and QueryNode raises TextIndexNotFound if the stats are not ready.

Two edge cases: BM25 output fields cannot be raw‑retrieved, while MinHash and TextEmbedding outputs can, because BM25 sparse vectors are retrieval‑only representations. Segment reuse also depends on function output: only segments whose manifest already contains all function‑output columns are reusable; older segments lacking new function columns are rebuilt.

2. Additive‑only schema refresh: patching the manifest

When a new column is added to an external table, existing external fragments still need a new manifest column group and a fake binlog. Keeping the old segment would make it lag behind the current schema, leaving the new field invisible to both manifest loading and memory‑estimation at query time. Refresh therefore handles this increment.

DataNode evaluates each current segment and classifies it into three outcomes:

Fragment missing → segment becomes invalid, remaining fragments become orphan.

All fragments present and all external fields covered by fake binlog ChildFields → kept (only ID is returned).

All fragments present but some external fields missing → patched (full SegmentInfo returned).

The key check is field coverage: the task schema extracts target external fields, the fake binlog ChildFields provide existing coverage, and missing fields are expressed using external column names because the manifest column group references source column names.

A patched segment follows a same‑ID patch flow: the same segment ID receives a new manifest and a new schema version while everything else stays unchanged. The process reads existing column groups, filters out already‑present columns (idempotent), creates new column groups, commits the new version, samples the size of the new fields, recomputes memory estimates, and rebuilds the fake binlog. The returned SegmentInfo only changes ManifestPath and SchemaVersion; segment ID, partition ID, insert channel, row count, state, and level remain.

DataCoord treats the patched result as an upsert payload: a non‑existent ID creates a new segment, an existing ID triggers a patch. Patch validation enforces two invariants—row count cannot change and schema version cannot roll back. The actual update touches only ManifestPath, SchemaVersion, Binlogs, and StorageVersion. The operation is atomic: drop non‑kept/updated segments, add the new segment, then patch the existing one.

The design document mentions a job/task‑level schema‑version gate to prevent race conditions between AddField and Refresh, but the current source code lacks this implementation. If an AddField races after the Refresh request is built, the task may finish with the older schema and skip the new field; a subsequent Refresh self‑heals via missing‑column detection.

User documentation confirms that external collections only support adding fields; dropping, renaming, type changes, remapping external_field, SPARSE_FLOAT_VECTOR, or StructArray fields are not supported.

3. milvus‑table: turning a snapshot into an external table

milvus-table

is the fifth external spec type (after parquet, lance-table, vortex, iceberg-table). Its external_source points to a snapshot metadata JSON file (must end with .json).

Reading requires a non‑empty storagev2_manifest_list in the snapshot. The system iterates the manifest list, converting each source StorageV3 segment manifest into a FileInfo. L0 segment deltalogs are collected as L0 overlays and attached to each fragment, forming a snapshot‑level delete overlay.

Field‑ID alignment is crucial. StorageV3 manifests store physical columns by field‑ID string (e.g., source field ID 101 → column 101). If the target collection generates different field IDs, reads would miss or mis‑map columns. RootCoord therefore aligns each user data field’s field ID with the source field ID; virtual PKs and function‑output fields receive target‑only IDs, and the property preserve_field_ids=true is written so that DDL replay does not need to reread the snapshot.

Two reasons force the source to be a normal StorageV3 collection: (1) only StorageV3 snapshots contain storagev2_manifest_list, and (2) external collections reject snapshots that are not from a regular StorageV3 collection (called “external‑table chaining”).

Delete handling differs by PK mode:

Real PK (target inherits source PK): reuse the source PK’s bloom filter for query pruning, reference source delete records without copying, and load the source insert‑timestamp column to preserve delete‑before‑reinsert visibility.

Virtual PK (target has no user PK): the bloom filter cannot be reused; DataNode translates source delete records into target virtual PK delete records by reading source deletes, locating matching rows, converting row numbers to virtual PKs, and writing them into the target’s delete log. The cost scales with the number of deletes, not total rows.

Refresh identity is defined as source_manifest_path:start_row:end_row. Delete logs do not participate in identity; therefore, a fragment that changes only by overlay updates keeps the same segment ID while the manifest is rewritten.

Putting the three mechanisms together yields a closed loop: internal collection → snapshot → milvus-table external source → External Collection. Batch jobs (e.g., Spark) can read the snapshot directly, while serving workloads query the same manifest‑backed view without copying data, providing a zero‑copy shared view.

4. Boundaries and author judgment

All three capabilities share a common thread: the manifest is the visibility commit point, and both external source columns and Milvus‑generated columns coexist within the same manifest. Function‑output fields grow a column on top of the source, additive‑only refresh patches the manifest, and milvus-table treats a snapshot’s manifest as an external source. None of them require copying the source table.

Community feedback (research date 2026‑08‑15) notes that no independent performance benchmark exists; the only deep‑dive analysis is from Elestio’s blog, which demonstrates a workflow of adding a column, using a snapshot as a consistent starting point, running offline embeddings, writing back, and incrementally building indexes—turning a weekend‑long migration into a hot‑path operation.

Open issues such as lakehouse integration (issue #45881) and schema editing support (discussion #35221) remain unimplemented in 3.0.0; the current release provides the implementation, not a roadmap. Future support for chaining (using an external collection’s snapshot as an external source) and a proper schema‑version gate will depend on demand for destructive schema changes like drop or rename. Until then, the self‑healing design of additive‑only refresh is considered sufficient.