Milvus 3.0 Upgrade Is Easy, Rollback Is the Real Danger

Changing the Helm image tag from 2.6.x to v3.0.0 is a lightweight action, and the official upgrade path describes it as compatible with rollback promises. However, the same documentation states that once Milvus writes data using Storage V3, downgrading to a version that cannot read Storage V3 is unsupported. These two statements are not contradictory — they address different things: the upgrade action versus the post-upgrade system state.

01 Don't Equate "Rollback" with Reverting the Image

Many 2.x operators intuitively follow a "backup — upgrade — issue — revert image" path. But whether a database can roll back depends not on the image tag alone, but on whether the old version can still interpret the current data, indexes, and WAL. Milvus 3.0's irreversibility boundaries are scattered across several layers: Storage V3 changes data layout, index versions change derived file layout, WAL message versions change encoding/decoding paths, and SDK/proto changes alter the access contract. These boundaries form at different moments.

For Storage V3 and index files, simply starting 3.0 does not automatically rewrite all existing layouts. The rollback window mainly narrows during new writes after upgrade, background compaction, or index rebuilds. Whether WAL messages undergo the same state change must be verified per message path; it cannot be inferred from a cross-layer generalization.

The core risk question becomes: Which write path first turns the system state into something the old version cannot interpret? This matters more than remembering an image tag.

02 Storage V3: The First Gate Opens After the First Write

Storage V3, backed by Loon FFI, is controlled by common.storage.useLoonFFI. Its source-code default is false, and the official docs confirm Storage V3 is disabled by default. Enabling it does not trigger an immediate full migration of existing data. The docs explicitly state: new writes and compaction output use Storage V3; existing data retains its original layout; during transition, the system can read both layouts.

This is the easiest point to misjudge. After only the image upgrade, old segments may still use the old layout, so the rollback window may not have closed yet. But once a new write lands as Storage V3 data, an old version that cannot read Storage V3 may fail to read that portion. Background compaction then rewrites qualifying existing segments to V3, expanding the affected data range.

The source-code targetVersion() makes this path direct: when useLoonFFI=true, compaction's target version is StorageV3; when disabled, the target returns to StorageV2. Note "target returns" — it only changes future compaction targets; it does not convert already-written V3 data back to V2. The official docs also warn that disabling Storage V3 neither immediately converts existing V3 data nor restores old-version compatibility.

An often-overlooked safety net: even with UseLoonFFI disabled, collections containing TEXT fields must remain on V3 to avoid data loss. The configuration appears as a toggle, but the data state is not a switch that can be freely flipped back.

Background compaction migration depends on scheduling, rate limits, and segment eligibility; "enabling the switch" cannot be equated with "all existing data has migrated." But before upgrading, you must treat it as a continuous state evolution, not a one-time action.

03 Indexes: Old Files Unchanged Doesn't Mean New Files Won't Cross the Line

The index layer's boundary is quieter. In milvus.yaml, targetVecIndexVersion defaults to 10, targetScalarIndexVersion to -1; forceRebuildSegmentIndex and forceRebuildScalarSegmentIndex default off; autoUpgradeSegmentIndex also defaults off. Release notes indicate new index versions require manual enabling (e.g., vector index version 10, scalar index version 4).

These configs tell us: Swapping to the 3.0 image does not mean all index files upgrade simultaneously. The version manager in source code combines target versions and cluster capabilities. QueryNodes report their supported version ranges; the cluster takes the minimum across all nodes to ensure rolling upgrades can load corresponding indexes. The final version is further clamped by clampVersion to the cluster's actual supported range.

Old indexes do not auto-rewrite just because the image changed. The actions that truly change file layout are new index creation, compaction-triggered rebuilds, or explicit force rebuild. compaction_trigger.go contains paths like "index version too old, trigger compaction" and "scalar index version != target, trigger compaction."

This gives index rollback risk a less dramatic but more troublesome shape: upgrade day may be fine, but after running a while, background tasks rebuild some old indexes to the new version. When you discover the issue and revert the image, the old engine faces index files it cannot read. storePathVersion comments are blunt: Layout 1 cannot be read by 2.6; enabling it commits the upgrade and forfeits rollback to 2.6; existing index files keep their original layout; the switch only affects subsequently built index files.

Therefore, the index layer must be split into two concerns:

Old indexes stay unchanged — image revert may still have room;

New indexes already built with the new layout, or old indexes already rebuilt — revert cannot rely on just swapping the image.

Index version is derived state. It may not cross the line when the upgrade command runs, but it can cross while the system continues working.

04 WAL Is Not a Version-Agnostic Tape

If Storage V3 changes data and indexes change derived files, WAL changes how messages are interpreted. Source code distinguishes message versions: VersionOld=0, VersionV1=1, VersionV2=2. Old-version messages come from the pre-streamingnode msgstream architecture; V1 encoding/decoding still uses msgstream; V2 encoding/decoding no longer depends on msgstream. Parsing paths are version-segregated; encountering an unknown message version causes a panic.

3.0 does not completely ignore historical messages. The WAL adaptor includes a V0-to-V1 conversion path for the streaming service to consume; source comments acknowledge this conversion has performance overhead, is only expected to handle small amounts of old messages, and unsupported message types will panic.

This compatibility layer is valuable but cannot be understood as "all WAL versions can replay each other." Message headers also carry storage_version; BatchUpdateManifest -related messages carry or handle manifest_version; V2 column groups are present. Messages describe not an abstract "data to write" but operations bound to storage version and Manifest state.

In short, WAL is not a format-agnostic tape. It stores semantic state changes.

Based on the source definition of V2 "no longer depends on msgstream," new and old consumption semantics are not simply interchangeable. But the specific read behavior of 2.6 against all V2 WAL messages is currently marked as pending verification and cannot be stated as a definite error or behavior.

This defines an operational boundary: historical old messages have a conversion path, but that does not mean new messages produced after upgrade can be fully interpreted by the old version. Whether to stop writes, drain or retain message backlogs must be validated against actual message paths and the target old version; "WAL still exists" cannot imply "rollback is definitely feasible."

05 SDK and Proto: Not Data Rollback Boundaries, But Day-One Contract Boundaries

The access layer carries different risks. 3.0 server depends on github.com/milvus-io/milvus-proto/go-api/v3; the Go client module path is github.com/milvus-io/milvus/client/v3. The 3.0.0 release notes list SDK versions: Python 3.0.1, Node.js 3.0.3, Java 3.0.5, Go 3.0.0. go-api/v3 and client/v3 are major version changes. They primarily affect how callers understand the API and proto, not the same kind of irreversible data-format boundary as Storage V3. The exact compatibility matrix between old SDKs and 3.0 server is a pending verification item.

But this doesn't mean client alignment can be postponed until problems appear. If the type contracts among server, client, and proto aren't checked together, the upgrade may first break at the access layer, not wait until the data-read phase. At that point you're rolling back the call contract, not reverting already-written data.

These four layers cannot be merged into a single "version compatibility" metric: Storage V3 concerns data layout, indexes concern file layout, WAL concerns message semantics, SDK/proto concerns interface contracts. Their failure moments and remediation differ.

06 Which Actions Can Revert, Which Can Only Roll Forward

Grouping common actions clarifies the boundaries.

Relatively revertible actions (usually those that haven't yet touched new state):

Only swapped the image, but no new Storage V3 writes occurred and no new-layout indexes generated;

Validated config in staging, confirming production data wasn't pushed to new layout;

Adjusted target versions before the cluster produced new index files;

On discovering SDK/proto contract mismatch, stop ingress traffic and adjust per verified client-version combinations.

The "relatively" cannot be omitted. Specific data, index, and WAL states still need verification.

Actions that, once occurred, generally only allow rolling forward or backup restore :

New writes have already produced Storage V3 data;

Compaction has already rewritten existing segments to V3;

New indexes have already been generated in layouts the old version cannot read;

WAL contains new message semantics unverified by the old version, and the rollback version cannot confirm its read path.

The official upgrade doc's failure handling path is to stop writes and perform backup restore, not merely revert the image tag. This is a plain reminder: if the goal is to restore to a complete state the old version can interpret, the recovery point must precede those irreversible write paths.

The claim "image can downgrade, so the system can roll back" is not accepted. The image is just executing code; system state lives in object storage, index files, and message streams. Code rolls back; state does not automatically follow.

07 Before Upgrading, Draw Your Own Rollback Window

In staging, don't just verify Pods start. Check around write paths:

Confirm current value of common.storage.useLoonFFI and which writes and compactions will adopt Storage V3 after upgrade.

Identify any TEXT collections. Even with Loon FFI disabled, related data may still be bound by the V3 safety net.

Record targetVecIndexVersion, targetScalarIndexVersion, storePathVersion, and verify all QueryNodes' supported index version ranges.

Observe whether compaction, index rebuilds, and new index generation occur after upgrade. Migration depends on actual scheduling conditions; don't replace state confirmation with a fixed time window.

Inventory WAL message versions and backlogs; validate message read paths against the target rollback version; V2 WAL behavior on 2.6 remains a pending verification item.

Align server, client, and proto major versions; verify old SDK compatibility with 3.0 server in your own deployment combination.

Prepare a backup recovery point earlier than new writes, and make "stop writes" part of the failure path, not a last-minute thought.

Scenarios truly fit for upgrade are those where the team can accept that certain write paths become forward-only after upgrade and has already validated the recovery path. Scenarios unfit for direct upgrade are those still defining rollback as a single Helm image change, or unable to confirm which data background compaction and index rebuilds have already pushed into the new state.

Milvus 3.0's problem isn't that the upgrade command is complex; it's that after the command ends, the system keeps writing, compacting, and rebuilding indexes. First answer which path rewrote your data, then decide whether to upgrade.