Choosing the Right Replication Factor for 10M QPS Message Queues: From Dual to Multi‑Replica

Real‑World Crash Illustrating Dual‑Replica Failure

At 2 am a broker handling core transaction traffic crashes; the primary stops sending heartbeats, the standby takes over, but a batch of messages that had not been synced in the last 200 ms is lost, causing downstream order mismatches.

Why One Replica Is Not Enough

Hardware failures include disk loss, power loss, NIC glitches, kernel panics, data‑center power cuts, and fiber cuts. RAID protects only disk failures; other failures still cause node loss. As cluster size grows from dozens of brokers at 1 M QPS to hundreds or thousands at 10 M QPS, absolute failure frequency increases by three orders of magnitude, making daily node outages inevitable.

Replication as Space‑for‑Availability

Replication trades storage space for higher availability and fault‑tolerance. The key decisions are number of replicas, placement, and synchronization protocol.

Dual‑Replica: Benefits and Two Major Pitfalls

Sync replication latency tax : The primary must wait for the standby’s acknowledgment. The write path includes local flush, network transfer, standby flush, and ack. Same‑data‑center latency is ≈2 ms; cross‑availability‑zone (AZ) latency rises to 5‑10 ms, which can halve throughput at 10 M QPS.

Async replication data‑loss risk : The primary returns after local flush while the standby catches up later. If the primary crashes before the standby has replicated pending data, those messages are lost. Semi‑sync (ack after in‑memory receipt) still loses data when both nodes fail simultaneously.

Split‑brain during failover : Determining whether a missing primary is truly dead or merely experiencing a network glitch requires an external arbiter (e.g., ZooKeeper, etcd). This adds a new failure domain, making overall availability dependent on the arbiter.

Three‑Replica: The First Sweet Spot

With three replicas, the system gains self‑election capability: if any two nodes acknowledge a write (majority), the remaining node can become the new leader without external arbitration. This is the core idea behind consensus algorithms such as Raft and Multi‑Paxos, implemented in Kafka’s KRaft, RocketMQ’s DLedger, and Pulsar’s BookKeeper.

Three‑replica also dramatically improves fault tolerance: losing one node still leaves a majority that has the data, and the write latency is only the time to sync with two nodes, not three.

The replica‑count vs. latency curve is U‑shaped. Too few replicas reduce availability; too many increase the “fastest‑two” tail latency because the write must wait for the slowest majority member. Empirically, three replicas cover 99.99 % of failure scenarios while keeping latency loss acceptable, which is why Kafka, Pulsar, and RocketMQ default to three.

Scaling Beyond Three: Cross‑AZ and Five‑Replica Deployments

When a service spans three AZs, each AZ must host a full broker cluster. Placing one replica per AZ ensures resilience against an entire AZ outage, but the majority write now requires an inter‑AZ round‑trip (≈5 ms), inflating overall latency.

Upgrading to five replicas raises the majority to three. The primary can ack after receiving confirmations from its own AZ plus any one replica in another AZ, reducing cross‑AZ latency to a single hop. The trade‑off is a 67 % increase in storage cost and higher operational complexity, but it protects core transaction paths from costly cross‑AZ failures.

Geo‑Distributed Multi‑Active Architecture

In an extreme case, each region (e.g., Beijing, Shanghai, Shenzhen) runs its own three‑replica cluster, totaling nine replicas. Regions replicate asynchronously to each other, achieving eventual consistency. If an entire region goes down, the other regions continue processing, with a brief window of a few seconds to minutes where some messages may be lost. This design sacrifices strong consistency for survivability.

Fine‑Grained Synchronization Strategies

ISR (In‑Sync Replica) : Kafka’s ISR requires a write to be acknowledged by all replicas currently in the ISR set, which dynamically excludes lagging nodes. The min.insync.replicas setting defines a safety floor; if the ISR falls below this number, writes are rejected.

Quorum configuration : Pulsar’s BookKeeper separates ensemble size, write quorum, and ack quorum, allowing writes to be persisted to three replicas while only waiting for two acks, decoupling durability from latency.

Read‑write replica separation : Writes go through the leader and ISR, while consumers can read from any replica. This boosts read throughput but may return slightly stale data, moving from strong consistency to “read‑your‑writes” or eventual consistency.

Replica Placement Engineering

Rack awareness : Ensure replicas of the same partition are on different racks to avoid rack‑level failures.

AZ awareness : Distribute replicas across AZs (e.g., three‑replica: one per AZ; five‑replica: two per two AZs and one in the third) so that a single AZ outage does not break the majority.

Hotspot awareness : High‑traffic topics should have their replicas spread across many physical nodes to prevent I/O saturation on a few machines. Systems like Pulsar’s Bundle Load Balancer and Kafka’s Cruise Control continuously rebalance replicas based on real‑time load.

Capacity Planning: Cost of Replication

Assuming 100 TB of data written daily and a 7‑day retention, storage grows linearly with replica count. CPU overhead does not double with more replicas because serialization/compression dominate, but network traffic scales linearly: each additional replica adds a full copy of internal traffic. In a 10 M QPS cluster, replication traffic often becomes the largest internal load, exceeding client‑facing traffic, so network architecture must be sized for replica traffic, not just external traffic.

Decision Logic for Choosing Replication Factor

Can the business tolerate data loss? → Minimum three replicas.

Is the business latency‑sensitive? → Limit replica count and avoid cross‑AZ majority writes.

Is the deployment multi‑AZ? → Use ≥3 replicas with AZ‑aware placement.

Is the deployment multi‑region? → Three replicas per region + asynchronous multi‑active across regions.

Evolution Summary

100 K QPS – single replica is sufficient.

1 M QPS – dual replica offers the best cost‑performance trade‑off.

10 M QPS – three replicas become the baseline, with cross‑AZ deployment as standard.

Multi‑region – add extra replicas and asynchronous multi‑active to achieve “service‑without‑interruption”.