Milvus 3.0 Streaming Architecture: 16 PChannels, Five Interceptor Layers, and a Self‑Built WAL 5.8× Faster Than Kafka

WAL‑first Architecture: Single Write‑Ahead Log

Milvus 2.x used two separate data paths: DML messages travelled Proxy → Pulsar/Kafka → DataNode and DDL messages travelled Proxy → etcd → RootCoord. The physical isolation prevented cross‑component consistency and made fault diagnosis difficult.

Milvus 3.0 merges all mutations—including Insert, Delete, Flush, CreateCollection, CreateIndex and RBAC—into a single WAL. The new components are:

StreamingCoord – assigns physical channels (PChannels) and coordinates atomic broadcast.

StreamingNode – runs the interceptor chain and performs WAL I/O.

StreamingClient – library embedded in Proxy and QueryNode, accessed via streaming.WAL().

This redesign provides a single source of truth and global visibility for every mutation.

Three‑layer Channel Model

The model defines three orthogonal channel types:

PChannel (Physical Channel) – the basic unit of parallelism. The default number is 16 ( rootCoord.dmlChannelNum) and each assignment increments a term for fencing.

VChannel (Virtual Channel) – collection‑level shards. Name format:

<pchannel_name>_<collectionID>v<shardIndex>

. Multiple VChannels can share the same PChannel, enabling multiplexing.

CChannel (Control Channel) – a permanent anchor bound to a specific PChannel. It carries a cluster‑wide ordering point via TimeTick messages.

Separating parallelism (PChannel), isolation (VChannel) and global ordering (CChannel) solves the 2.x problem where DML and DDL were isolated and could not be made atomically visible.

Five‑layer Interceptor Chain (Onion Model)

When a StreamingNode receives an Append request it passes the message through five interceptors, each handling a single responsibility. The registration order (outermost first) is:

[]interceptors.InterceptorBuilder{</code>
<code>    redo.NewInterceptorBuilder(),      // 1. retry handling</code>
<code>    lock.NewInterceptorBuilder(),      // 2. locking</code>
<code>    replicate.NewInterceptorBuilder(), // 3. CDC replication</code>
<code>    timetick.NewInterceptorBuilder(), // 4. timestamp allocation</code>
<code>    shard.NewInterceptorBuilder(),    // 5. segment allocation</code>
<code>}

Key behaviours of each layer:

redo – retries recoverable errors (e.g., outdated TimeTick, segment not ready, fenced) before acquiring locks.

lock – maintains a PChannel‑level sync.RWMutex (glock) and a VChannel‑level key lock. ExclusiveRequired messages acquire exclusive locks; DML messages use shared locks.

replicate – processes CDC replication. AlterReplicateConfig messages trigger mode switches and optional rollbacks; other messages are passed through unchanged.

timetick – allocates a TimeTick from the TSO, tracks three visibility states (Allocated, Confirmed, Synced) and generates a non‑persisted TimeTick for empty batches to avoid unnecessary I/O.

shard – assigns segments via shardManager.AssignSegment() and applies one of 13 seal policies (capacity, idle, growing, etc.).

Broadcaster: Atomic Cross‑PChannel Broadcast

DDL operations must be atomically visible on all PChannels. The Broadcaster (a sub‑module of StreamingCoord) implements a six‑step workflow:

Lock – acquire ResourceKey locks in a deterministic order.

Persist – allocate a BroadcastID and persist a PENDING task to the catalog.

Append – write the broadcast message to every target PChannel via AppendMessages().

FastAck – DDL can self‑acknowledge without waiting for consumer ACKs (unless AckSyncUp is set).

AckCallback – callbacks are executed in CChannel TimeTick order, guaranteeing global ordering for conflicting ResourceKey tasks.

Tombstone & GC – the task is marked TOMBSTONE and later cleaned up.

Locks are based on Resource Keys rather than raw PChannels, e.g., NewSharedCollectionNameResourceKey(db, coll) for collection‑level shared locks and NewExclusiveClusterResourceKey() for cluster‑wide exclusive locks. This unifies atomic broadcast with collection‑level concurrency control.

Woodpecker: Self‑Built WAL and Performance Gains

Milvus replaces external Kafka/Pulsar with its own WAL implementation called Woodpecker. The motivations are:

Operational complexity – external MQs require dedicated nodes, ZooKeeper/BookKeeper and increase the failure surface.

Architectural constraints – limited topic count forced the VShard workaround, adding complexity.

Resource inefficiency – transient signals such as TimeTick were unnecessarily persisted.

Woodpecker’s Zero‑Disk architecture stores log data in object storage (S3, GCS, MinIO) while metadata lives in etcd, decoupling compute from storage.

Performance benchmark (official blog) compares throughput and latency:

Throughput – Kafka 129.96 MB/s, Pulsar 107 MB/s, Woodpecker WP MinIO 71 MB/s, Woodpecker WP Local 450 MB/s , Woodpecker WP S3 750 MB/s .

Latency – Kafka 58 ms, Pulsar 35 ms, Woodpecker WP MinIO 184 ms, Woodpecker WP Local 1.8 ms , Woodpecker WP S3 166 ms.

Key takeaways: Woodpecker is 3.5× faster than Kafka and 4.2× faster than Pulsar on local disks; in S3 mode it is 5.8× faster than Kafka and 7× faster than Pulsar. The local mode achieves single‑digit millisecond latency suitable for financial‑grade workloads.

Deployment modes:

MemoryBuffer – caches writes in memory and flushes to object storage; suited for small batch‑oriented deployments.

QuorumBuffer – writes with a 3‑replica quorum, delivering sub‑millisecond latency; suited for finance‑grade scenarios.

Woodpecker can run in Embedded Mode (library inside Milvus) or Service Mode (standalone LogStore cluster). Compatibility matrix (conditional write support) includes AWS S3, Azure Blob, MinIO ≥ 2024‑12, Alibaba OSS, Tencent COS, GCS, VAST Data; Huawei OBS is not supported.

Pluggable WAL Back‑ends and AlterWAL

Milvus 3.0’s WAL back‑end is pluggable and supports four implementations:

Woodpecker – native Milvus WAL, recommended for new clusters.

Kafka – generic MQ, used when existing Kafka infrastructure exists.

Pulsar – generic MQ, used when existing Pulsar infrastructure exists.

RocksMQ – in‑process RocksDB queue, used for standalone single‑node deployments.

Selection priority (from configs/milvus.yaml) is:

Standalone: rocksmq > Pulsar > Kafka > Woodpecker Cluster: Pulsar > Kafka > Woodpecker AlterWAL messages enable runtime WAL back‑end switching without downtime. The switch proceeds in two phases:

FLUSHING – flush all growing segments.

ADVANCE_CHECKPOINT – update VChannel and PChannel checkpoints to the new WAL position.

Milvus 3.0‑beta upgrades still forbid changing the MQ type; the mechanism is ready for future releases.

RecoveryStorage: Proven Fault Recovery

RecoveryStorage guarantees that from any WAL position plus persisted state the system can replay forward to reconstruct a consistent in‑memory state. The invariant is:

From any WAL position + corresponding persisted state, RecoveryStorage can forward‑replay the WAL to fully recover the memory state.

RecoverySnapshot

contains VChannel metadata, segment assignments, checkpoint, transaction buffer and optional AlterWAL info.

Recovery proceeds in two steps: recoverRecoveryInfoFromMeta() – loads checkpoint, VChannel metadata and segment assignments in parallel from the catalog. recoverFromStream() – builds a RecoveryStream from the checkpoint to the current WAL position and replays messages to rebuild memory.

This design ensures that any StreamingNode restart restores a consistent state without manual intervention, unlike the fragmented 2.x recovery process.

Transaction System

Milvus 3.0 introduces a true transaction system with message types:

BeginTxn = 900

CommitTxn = 901

RollbackTxn = 902

Txn = 999

TxnManager

tracks in‑flight transactions per VChannel and provides operations: BeginNewTxn() – allocate TxnID and TimeTick, create a TxnSession. FailTxnAtVChannel(vchannel) – interrupt all transactions on a VChannel (called by the lock interceptor for exclusive messages). RollbackAllInFlightTransactions() – invoked when a force‑promote DDL aborts conflicting DML. GracefulClose() – wait for all transactions to finish before shutdown.

Transaction lifecycles have six possible outcomes: Begin, Append, Commit, Rollback, Expire (TTL), and Force‑fail (used by DDL to abort conflicting DML).

Known Incompleteness

End‑to‑end idempotency is still a work‑in‑progress (Milvus issue #50000).

3.0‑beta cannot switch MQ back‑ends during upgrade.

The interceptor order is inferred from source code; no official RFC explains the design rationale.

Internal documentation contains 19 markdown files, but public release notes lack detailed coverage of the streaming system.

Conclusion

Milvus 3.0 rewrites the write path from a fragmented dual‑track architecture to a WAL‑first, globally ordered design. The combination of the three‑layer channel model, five‑layer interceptor chain, atomic Broadcaster, Woodpecker WAL, pluggable back‑ends, AlterWAL and RecoveryStorage provides structural consistency, high performance, simplified operations and provable fault tolerance.