Why Milvus 3.0 Needed a Complete Storage Rewrite: From Scattered Binlogs to a Manifest Table

Storage version break

In data_coord.proto a comment states that when storage_version >= 3 binlog paths must be resolved from manifest_path. This marks a hard break from Milvus 2.x, which stored each column in separate binlog files, to Milvus 3.0’s self‑describing manifest table format implemented by the new storage engine Loon .

1. Three generations of storage and their technical debt

V1 – Per‑binlog file format

What it solved : Simple implementation sufficient for early Milvus 2.0 usage.

Debt : Hundreds of tiny files per segment cause heavy metadata pressure on object storage; no transactional semantics, so crashes can leave partially written files.

V2 – Packed writer + Parquet

Replaces the custom binlog with Parquet and packs multiple columns of a segment together, reducing file count and improving compression.

Metadata (file‑to‑segment mapping and snapshot state) remains in DataCoord’s in‑memory structures, not persisted as a self‑describing table.

V3 – Loon manifest format

Introduces a full table‑format design: an Avro‑encoded manifest anchors all metadata (column groups, delta logs, stats, indexes). Scanning the manifest reveals the complete state of a table.

const (
    StorageV1 int64 = 0 // milvus 2.0 legacy binlog format
    StorageV2 int64 = 2 // milvus‑storage packed writer (Parquet)
    StorageV3 int64 = 3 // loon manifest format
)

2. Manifest table format – Iceberg‑like layout

The directory layout mirrors Apache Iceberg:

base_dir/
├── _metadata/manifest-{version}.avro   # Avro, version starts at 1
├── _data/{group_id}_{uuid}.{format}    # Parquet, Vortex, Lance, binary
├── _delta/                             # Deletion logs
├── _stats/                             # Bloom filter, BM25, etc.
└── _index/                             # HNSW, IVF, bitmap, …

Each manifest starts with magic number 0x4D494C56 (ASCII "MILV") followed by an int32 version field. Version 1 stores column_groups + delta_logs + stats; version 2 adds an indexes field. Version 1 manifests can be read by version 2 code, but forward compatibility is not guaranteed.

Go‑side lightweight encoding

Go only needs the manifest path and passes a minimal JSON structure to the Rust engine:

type ManifestJSON struct {
    ManifestVersion int64  `json:"ver"`
    BasePath        string `json:"base_path"`
}

// Example JSON: {"ver":1,"base_path":"/path/to/manifest"}

Design principle: Go coordinates, Rust does the I/O work .

3. Transaction system – optimistic lock with three resolvers

FailResolver

: if read_version != seen_version, fail immediately (strict serialization). MergeResolver: merge concurrent changes into seen_manifest. OverwriteResolver: apply updates to read_manifest ignoring concurrent changes.

The default is OverwriteResolver because stats updates use key‑wise overwrite semantics, while column‑group additions and delta‑log appends use orthogonal key spaces, making overwrite safe and simple.

Automatic index eviction

When AppendFiles adds new data to a column group, the old index for that column is automatically dropped, preventing stale indexes from misleading queries.

Retry mechanism

Conflicts are retried up to commonCfg.ManifestTransactionRetryLimit (default 10). The comment explains that multiple stats tasks may concurrently write the same segment’s manifest; on conflict the transaction reads the latest version and reapplies its updates.

4. Column‑group splitting – access‑pattern isolation

V3 introduces ColumnGroup with a Format field, allowing each group to declare its own file format (Parquet, Vortex, Lance, or binary). The splitting pipeline consists of four ordered policies:

SystemColumnPolicy – configurable split of PK, partition key, clustering key.

AvgSizePolicy – columns larger than a threshold (default 1024 B) are isolated.

SelectedDataTypePolicy – always splits vector and TEXT columns into dedicated groups.

RemanentShortPolicy – merges remaining small columns into a default group.

The core idea is to isolate columns by their typical access patterns: point‑lookup columns, vector similarity columns, and TEXT columns are read independently, avoiding unnecessary I/O.

5. Go ↔ Rust FFI boundary – responsibilities

Milvus’s main process (Go) delegates all I/O and encoding to the Rust Loon engine via a C ABI. Four FFI function chains illustrate the interaction:

Packed Writer : loon_writer_new → loon_writer_write → loon_writer_close (returns a C pointer to ColumnGroups).

Segment Writer (handles TEXT LOB):

loon_segment_writer_new → loon_segment_writer_write → loon_segment_writer_flush → loon_segment_writer_close

(returns SegmentOutput containing column groups and LOB files).

Transaction :

loon_transaction_begin → append_files / add_column_group / add_delta_log / add_lob_file → loon_transaction_commit

.

Reader : NewPackedFFIReaderWithManifest → GetFFIReaderStream (manifest path is passed to Rust for column‑group resolution).

A TODO comment in ffi_common.go notes that all Loon FFI failures are currently treated as transient errors ( ErrLoonTransient), meaning even unrecoverable I/O errors are retried up to ten times – a marked technical debt.

6. TEXT LOB mixed storage – why not Parquet

Milvus 3.0 adds a TEXT field type for full‑text search. Parquet is unsuitable because:

Columnar compression for long strings is mediocre.

Random access requires scanning an entire row group (O(N)).

The solution is an inline + LOB hybrid. Each TEXT value starts with a 1‑byte flag: 0x00 – inline short text stored directly in the column‑group file. 0x01 – LOB reference (24 B) pointing to a separate Vortex file stored at the partition level, allowing multiple segments to share the same LOB.

LOB files are placed under the partition directory, e.g.:

{root}/insert_log/{collection}/{partition}/
├── {segment}/
│   ├── _metadata/manifest-{version}.avro
│   └── _data/cg*.parquet   # column‑group files (one may contain TEXT refs)
└── lobs/
    └── {field_id}/_data/{file_id}.vx  # Vortex format

Compaction strategies depend on the hole_ratio (percentage of deleted LOB references): hole_ratio < 0.3 → REUSE_ALL : few holes, keep existing LOB files. hole_ratio >= 0.3 → REWRITE_ALL : many holes, rewrite dense LOB files.

Clustering compaction → REWRITE_ALL (reordering requires new LOB layout).

Sort compaction → REUSE_ALL (only order changes).

L0 delete compaction → SKIP (pure deletions, no LOB impact).

Rejected alternatives include managing file_id via etcd, removing the inline path entirely, placing LOB files inside the segment base path, and using Parquet for LOB files – each dismissed for performance or compaction‑reuse reasons.

Vortex was chosen for LOB files because it offers better long‑string compression and O(log N) split‑index random access compared to Parquet. The configuration key dataNode.storage.format can switch between parquet and vortex for regular column groups, but LOB files always use Vortex (extension .vx).

7. Migration from V2 to V3 – compaction driven

When the flag common.storage.useLoonFFI (default false) is enabled, new segments are written with storage_version = StorageV3. Existing V2 data is migrated gradually by compaction, which reads V2 segments, rewrites them using the manifest format, and produces V3 segments.

Three protection mechanisms guard the migration:

TEXT protection : collections containing TEXT fields cannot downgrade below V3.

Rate limiting : StorageVersionCompactionRateLimitTokens=3 limits upgrades to three per 120 seconds, preventing I/O saturation.

Session version gating : migration pauses if any QueryNode runs an older version, ensuring older nodes never read V3 segments.

External collections are exempt because their data resides outside Milvus storage.

8. Unresolved issues

FFI error codes are missing; all failures are currently treated as transient, causing unnecessary retries.

Vortex format lacks public documentation and performance benchmarks.

The TEXT LOB design document is still in Draft status, so some parameters may change.

Milvus‑storage has a small open‑source community (≈44 stars), indicating limited external contributions.

9. Why this matters

Milvus 3.0’s external collection feature, zero‑copy Spark reads, and full‑text search all rely on the Loon manifest table. Without it, virtual segments could not reference lakehouse splits, column‑group isolation would be lost, and TEXT LOB performance would degrade.

The storage rewrite is the foundation for Milvus’s evolution from a pure vector database to a Vector Lakehouse. Understanding this layer simplifies later topics such as V3‑based compaction, segment lifecycle, and index format design.

}