Scaling Message Queues to 10M QPS: From Downtime to Seamless Online Expansion

1. The Red Button That Once Worked

Ten years ago, at the 100k QPS level, a broker cluster typically consisted of only a handful to a few dozen nodes, and each broker held a modest number of partitions. Expansion was essentially "treat data as a resettable state and clients as tolerant of reconnections".

Pressing the red button meant choosing a low‑traffic window—usually 02:00‑04:00 am—when no production traffic arrived. Engineers could relax because metadata changes would not trigger cascades and partition reassignment would not cause producer send timeouts.

The cost was visible: business impact, overloaded support lines, and reconciliation work. As long as the scale stayed below a certain line, product managers could approve the cost.

1. Why It Worked Before

Business tolerated minute‑level message delay; brief disconnections could be caught up later.

Cluster size was small, so operations completed within tens of minutes.

Client fault tolerance was coarse; simple reconnect logic allowed continued processing after recovery.

In short, downtime expansion was not a technical choice but an engineering compromise that relied on "business pause" as a universal key.

2. When the Red Button No Longer Works

When the business moved from tolerating minute‑level delays to requiring sub‑second jitter reporting, when the cluster grew from dozens to hundreds of brokers, and when downstream services expanded from a few internal services to thousands of micro‑services, the cost of downtime grew exponentially:

Coordinating dozens of business owners for a single outage.

Data to migrate grew from hundreds of GB to hundreds of TB.

Large client fleets caused a connection storm after restart, becoming a failure source itself.

At this point, the cost of pressing the red button exceeded the cost of a failed expansion, forcing a shift from downtime to online expansion.

2. What "Online Expansion" Really Means

"Online expansion" is interpreted differently: some consider it successful if writes never drop to zero, others require immediate partition balance. To clarify, the expansion process is split into three layers—control plane, data plane, and client plane—each of which must stay online.

1. Control‑Plane Online: The Often‑Overlooked Risk

When a new broker joins, the cluster metadata (node list, partition‑to‑broker mapping, replica topology) must be updated. In ZooKeeper‑based systems this was a global write; in KRaft, BookKeeper metadata, or NameServer implementations each has its own consistency constraints.

Although the control‑plane update looks small, slow or brief metadata broadcast splits can give clients stale routing tables, leading to rejected messages or even split‑brain scenarios. In large clusters, metadata propagation shifts from sub‑millisecond to hundreds of milliseconds, and this window must be accounted for.

2. Data‑Plane Online: The Costliest Layer

The biggest controversy is whether historical data must be moved to new nodes. Two mainstream approaches exist:

For a million‑QPS cluster with evenly decreasing write hotness, a "no‑move" strategy like RocketMQ’s is sufficient.

For a ten‑million‑QPS cluster with hotspot partitions and long‑term data retention, data migration becomes unavoidable.

3. Client‑Plane Online: The Ecosystem Test

After the control and data planes are handled, the question is whether clients need to pause.

Producers can simply pick up the new topology on the next send.

Consumers are the pain point: partition‑to‑consumer assignment changes trigger a rebalance. Classic eager rebalance stops the whole consumer group for seconds to tens of seconds.

At ten‑million‑QPS, a one‑second pause would cause millions of messages to pile up. The final mile of online expansion is therefore a protocol upgrade from eager rebalance to cooperative (or sticky) rebalance, moving only the partitions that truly need to change.

3. Three Hard Barriers: Metadata, Data, Traffic

Each of the three layers above maps to a recurring real‑world failure point.

1. Metadata: The "Half‑Dead" Broadcast Window

When a new broker registers, the controller updates the partition assignment and pushes the new plan to all brokers and clients. In a hundred‑node cluster this push is instantaneous; in a few‑hundred‑node cluster each full push can be dozens of megabytes, creating a small traffic storm.

During expansion the cluster exists in a "new‑old routing coexistence" state. If not designed carefully, some clients may send to partitions that appear to exist but have already been removed, generating massive error logs and retries. The usual mitigation is a two‑phase commit: first ensure all brokers have fully synchronized metadata, then open writes—"prepare the backend before releasing the frontend".

2. Data: Migration Is Traffic

Moving historical data creates a huge internal traffic flow. An unrestricted hundred‑TB migration can consume dozens of Gbps of internal bandwidth, competing with production traffic for the same NIC, disk, and page cache.

If not throttled, migration can push the cluster’s P99 latency to extreme levels and cause client timeouts. Throttling is dynamic:

During business low‑peak periods, migration can be aggressive; during high‑peak, it must be restrained.

Hot partitions must not be moved during peak.

Network topology (cross‑rack, cross‑AZ) has different costs.

Mature implementations provide leader/follower replication throttling parameters and a whitelist mechanism that only limits the replica channels currently being migrated, avoiding impact on normal replication.

3. Traffic: Client Perception Cost

Even if brokers are fully online, the expansion fails if clients perceive a disruption. Consumer rebalance is the biggest pain point. Under the eager protocol, a rebalance stops the whole group, potentially for more than ten seconds in large groups. Cooperative rebalance splits the action: first release only the partitions that need to change, letting the rest continue consuming, then take over the new partitions. This turns a hard stop into a latency jitter.

4. Partition Rebalance: What to Move and When

The hardest part of online expansion is the partition rebalance algorithm, which answers two questions: which partitions to move, and in what order and speed?

1. Partition Selection: Minimize Movement

A naive round‑robin redistribution would cause many unnecessary moves. For example, expanding from 10 to 12 nodes ideally requires each node to give up only a small fraction of partitions, but a fresh plan could relocate up to 80 % of partitions. The mainstream solution introduces a "sticky" constraint: keep the existing distribution as much as possible while still achieving balance. Mathematically this is a constrained linear program: Goal: keep per‑node partition count and leader count as equal as possible. Constraint: preserve the current partition distribution. Weight: avoid placing replicas of the same rack together.

2. Move Order: Cold First, Hot Later

Even after selecting partitions, the order matters. Moving cold partitions first reduces online impact because they hold less data; once they finish, the algorithm converges partially. Hot partitions are moved during low‑traffic windows to avoid business peaks. In practice, operators also: Batch partitions so each batch stays within the internal bandwidth budget. Leave observation windows between batches to monitor P99 latency. Automatically pause the next batch if P99 exceeds a threshold.

3. Move Speed: Throttling Is Not "Higher Is Better"

New engineers often raise throttling limits to finish migration quickly, but this can cause hot partitions to time out as the whole consumer group stalls. Dynamic throttling—adjusting migration speed based on broker CPU, network, and disk usage—is a sign of engineering maturity. At million‑QPS it is a bonus; at ten‑million‑QPS it is mandatory.

5. Different Message‑Queue Expansion Paradigms

Although all three systems perform "expansion", Kafka, RocketMQ, and Pulsar follow three distinct engineering paths.

1. Kafka: Partition‑Level Reassignment

Kafka requires an explicit reassign‑partitions step after adding brokers. The reassignment is controllable, can be dry‑run, interrupted, or batched, but demands tooling for planning, throttling, and monitoring. At ten‑million‑QPS, Kafka’s expansion involves a full toolchain: automatic sticky reassignment generation, integrated throttling and batching, real‑time monitoring of migration rate versus business P99, and automatic pause/rollback on anomalies.

2. RocketMQ: Incremental‑Friendly, No‑Move for Existing Data

RocketMQ prefers operational simplicity: new brokers automatically receive new topics or queues, while existing topics stay where they are. This makes the expansion cost near‑zero—just register and adjust routing. The downside is that existing nodes do not shed load; if an old node is already a bottleneck, expansion yields limited benefit. Common mitigations include adding new queues for old topics on the new broker, gradually shifting producer traffic, and actively migrating hot‑topic historical data.

3. Pulsar: Bundle‑Level Routing Switch

Pulsar separates compute (broker) from storage (BookKeeper). Adding a broker triggers the load manager to move bundles (hash shards) to the new broker. No actual data moves; only ownership changes, completing in minutes. Advantages: data plane stays untouched, compute and storage scale independently, and client lookup automatically redirects. Drawbacks: higher architectural complexity, BookKeeper still needs ledger redistribution, bundle switch can cause brief jitter, and operators must understand two‑layer scheduling.

6. Devilish Details at Ten‑Million‑QPS

1. Metadata Weight Becomes Non‑Negligible

Below a hundred nodes, metadata broadcast is millisecond‑scale. At several hundred to a thousand brokers and tens of thousands of partitions, metadata can be dozens of MB, and each full push becomes a small traffic storm. Mitigations: Introduce incremental update protocols (diff instead of full). Physically separate control‑plane and data‑plane (dedicated NIC/VLAN). Clients use local cache and watch mechanisms instead of periodic pulls.

2. Connection Storm from Massive Client Scale

Ten‑million‑QPS back‑ends often have hundreds of thousands of client connections. A metadata update or rebalance can trigger a simultaneous reconnection burst, stressing CPU and file‑descriptor limits. Countermeasures: Introduce jitter on the client side to spread reconnections over a window. Configure broker‑side connection throttling and priority queues. Upgrade clients to support incremental metadata and cooperative rebalance.

3. Cross‑AZ "Amplification Effect"

Large clusters span multiple availability zones. New replicas pulling data across AZs generate expensive cross‑AZ traffic, turning migration bandwidth into a cost factor (tens of thousands of dollars per large reassignment). Best practices: Replica placement aware of rack/AZ, preferring intra‑AZ follower‑to‑follower sync. Throttle parameters per link type (same rack, same AZ, cross AZ). Scheduling algorithm prefers same‑AZ target brokers for data moves.

4. Shadow‑Traffic Warm‑Up

New brokers start cold: empty page cache, no open file handles. If they receive production traffic immediately, their P99 latency spikes above existing nodes, causing jitter on the expansion day. Warm‑up strategy: before exposing the broker to real traffic, use replica sync, synthetic load, or stress tests to heat up kernel caches, JIT compilation, and connection pools. This turns the risk from "Can the new node handle traffic?" to "The new node is already hot; we only need to switch traffic."

5. Rollback Plan: Expansion Must Be Reversible

At ten‑million‑QPS there are no "small actions"; each expansion is treated as a failure‑drill. The core rollback step is to snapshot metadata (topic/partition/replica layout) before any partition migration begins, allowing a full revert at any stage. Snapshot dimensions include: Metadata snapshot. Throttling parameter snapshot. Client version and configuration snapshot. Monitoring baseline (pre‑migration P99/TPS/lag). A complete rollback capability increases the team’s confidence to perform larger expansions.

7. Evolution Roadmap from Downtime to Online

From 100k QPS to 10M QPS, expansion evolved through a clear roadmap:

1. 100k QPS – Downtime Is Economical

Small cluster, simple clients, business tolerates brief interruption. Downtime expansion is the cheapest overall solution; no technical debt needs to be repaid.

2. 1M QPS – Begin Online‑ization

Business becomes latency‑sensitive, client count rises, downtime cost climbs. Rolling expansion, partition reassignment, and cooperative rebalance start appearing in the toolchain, but operations still aim for simplicity.

3. 10M QPS – Online Expansion Becomes Fundamental

Migration traffic, client connection storms, and cross‑AZ costs must be solved. A full toolchain emerges: automatic plan generation, dynamic throttling, shadow warm‑up, and rollback snapshots. Expansion capability itself becomes part of the SLA.

4. Beyond – Elastic Scheduling

At even larger scales, expansion turns from an operational task into a scheduling strategy. Clusters gain elastic ability to auto‑scale based on traffic; compute‑storage separation further lowers migration cost; multi‑tenant scenarios enable fine‑grained resource scheduling. This is the direction of cloud‑native message queues.

8. Back to the Red Button

Eight days before the peak, the business experienced a completely seamless traffic surge; the post‑mortem revealed that every engineering step—incremental metadata push, dynamic throttling, cooperative rebalance, shadow warm‑up, cross‑AZ routing, and rollback snapshots—was not about merely adding machines. It transformed the act of adding machines from a business‑level degradation into an operations‑level event. This mirrors the broader trend from million‑ to ten‑million‑QPS: turning actions that once required company‑wide coordination into tasks that a couple of engineers can complete calmly during the day. The larger the scale, the quieter the operations must be. Open questions for readers:

Which layer—metadata, data, or client—poses the biggest bottleneck for your expansion?

If you designed the next‑generation queue expansion protocol, which "devil‑in‑the‑details" would you prioritize?