Milvus 3.0 Snapshot: Pausing a Collection Without Copying Data

1. Snapshot Is Not a Backup

The official documentation distinguishes Snapshot from Backup: Backup creates an independent full copy of all data files for long‑term disaster recovery, while Snapshot targets day‑to‑week rapid rollback and versioning by persisting only metadata and manifest references to existing segment, binlog, and index files.

Because no data files are duplicated, the storage cost of a Snapshot is low, but any segment or index referenced by a Snapshot is protected from garbage collection (GC) until the Snapshot is dropped and the references are no longer present in any live collection.

2. Where the Point‑in‑Time View Comes From

Each channel in a collection has its own MsgPosition (seek timestamp). The Snapshot’s create_ts is the minimum of these timestamps and is used only for display and sorting; it is not a strict global boundary.

During creation, DataCoord iterates over non‑dropped segments and includes those whose channel’s seek position falls within the calculated boundary. The resulting view is a set of channel boundaries plus a list of referenced segment files, which the author describes as a “point‑in‑time read‑only view” rather than a generic MVCC snapshot.

3. Why Snapshot Creation Does Not Force a Flush

The management guide recommends stopping writes and calling flush() before creating a Snapshot, but it is not mandatory. If flush() is skipped, the Snapshot contains only already flushed data.

Source code shows that snapshotManager.CreateSnapshot() invokes handler.GenSnapshot(ctx, collectionID) to generate snapshot data and then persists it with snapshotMeta.SaveSnapshot(ctx, snapshotData). The design deliberately avoids interfering with the write path, keeping Snapshot creation lightweight and fast.

4. Manifest Is the Ledger of the Design

Milvus stores Snapshot information in two layers: etcd holds a SnapshotInfo for quick listing, while object storage contains a directory structure:

snapshots/{collection_id}/
├── metadata/
│   └── {snapshot_id}.json
└── manifests/
    └── {snapshot_id}/
        └── {segment_id}.avro

SnapshotWriter.Save()

first writes an Avro manifest for each segment, then records the storage‑V2 manifest paths, and finally writes the metadata JSON, which includes fields such as SnapshotInfo, Collection, Indexes, ManifestList, SegmentIds, BuildIds, and timestamps like StartPosition, DmlPosition, CommitTimestamp, etc.

The manifest version evolves (V0‑V4), adding fields like index_store_path_version, commit_timestamp, child_fields, and format to maintain backward compatibility.

5. Restore Uses Copy‑Segment Instead of Import

Restore copies the original segment files (binlogs, deltalogs, index files) directly, preserving field IDs and index IDs, which avoids data rewriting and index rebuilding. The workflow involves Proxy (entry), RootCoord (collection orchestration), DataCoord (creates copy‑segment jobs), and DataNode (executes the actual file copy).

The CopySegmentJob records details such as IdMappings, SnapshotName, SourceCollectionId, PinId, TotalSegments, CopiedSegments, and TotalRows. The job builds a CopySegmentRequest, DataNode copies files and generates new binlog paths, and DataCoord updates segment metadata before marking the target segments as flushed.

Restore must preserve field and index IDs ( PreserveFieldId, PreserveIndexId) because the internal files rely on these identifiers; otherwise the new collection cannot interpret the old files.

6. GC and Pin Protection Are the Hard Parts

When a Snapshot references a segment or index file, Milvus disables GC for those files until the Snapshot is dropped. The code checks snapshotMeta.IsSegmentGCBlocked(collectionID, segmentID) and IsBuildIDGCBlocked to perform O(1) lookups of protected references.

Pin/Unpin provides a temporary lease for background processes reading a Snapshot. PinSnapshotData returns a pin ID that prevents GC, and UnpinSnapshotData releases the lease. If a process crashes, a TTL fallback ensures eventual cleanup.

This mechanism guarantees that a Snapshot cannot be dropped while a restore or export is still reading its data, preventing premature deletion of required files.

7. Evolution: Snapshot as a System Constraint

Snapshot creation follows a two‑phase commit: a PENDING entry is written to the catalog, manifests and metadata are stored in object storage, then the state transitions to COMMITTED. This atomic flow allows GC to treat orphan files correctly on failure.

Restore assembles a series of background tasks (metadata restoration, index restoration, copy‑segment job, DataNode file copy, DataCoord metadata sync). The CopySegmentTask progresses through states PENDING → InProgress → Executing → Completed/Failed → Cleanup, with retention controlled by DataCoordCfg.CopySegmentTaskRetention.

Overall, Snapshot adds a historical view layer that forces the entire system—metadata, manifest, GC, and pin logic—to acknowledge and protect past references, enabling low‑cost snapshots while shifting consistency responsibilities to metadata management.

Summary

Milvus 3.0 Snapshot freezes a collection’s metadata and segment manifests into a point‑in‑time read‑only view, restores it via Copy‑Segment, and relies on GC/pin protection to keep referenced files alive. It is not a traditional backup; instead, it provides fast, lightweight rollback by protecting historical references throughout the system.