Cross‑Data‑Center Replication at Ten‑Million QPS: From Async to Semi‑Sync and How to Choose

Why Cross‑Datacenter Sync Matters

Before designing a solution, teams must ask what risk they are protecting against. While multi‑replica setups handle process or node failures, only cross‑datacenter sync can survive rack‑level, region‑level outages such as a cut fiber, power loss, or cooling failure. Compliance in finance and government also mandates off‑site disaster recovery.

Asynchronous Replication: Simple but Has a Window

In the earliest designs, producers receive an ACK as soon as the local datacenter writes the message; the copy to the backup site proceeds independently. This yields low latency (no extra RTT), high throughput, and clear decoupling, but introduces a permanent data gap (RPO) that can grow from 100 ms to seconds under load spikes. The article cites a real incident where a core switch failure left 470 k messages unsynchronized, causing direct financial loss. Monitoring replication lag can warn of large windows but cannot rescue messages already queued.

Strong Synchronous Replication: Safe but Latency‑Heavy

Strong sync requires both primary and backup sites to write successfully before ACK, achieving RPO = 0. However, each message incurs an extra cross‑datacenter RTT (30‑100 ms), throttling producer QPS (e.g., a 200 QPS producer drops to ~12 QPS across regions) and demanding many more producer instances. The approach tightly couples the business path to the network, making it vulnerable to jitter; a 5‑second network glitch can drop throughput to zero. Consequently, strong sync is rarely used outside high‑value financial transactions.

Semi‑Synchronous Replication: The Art of Compromise

Semi‑sync acknowledges a message after local replicas succeed and at least one copy is already in transit to the backup site, using a durable sync queue. Unlike pure async, unsynchronized messages are stored reliably, so a primary crash does not lose them. Some implementations add a "dual‑write" queue that also writes to a lightweight third‑AZ queue for extra safety. Advantages include controllable latency (near‑async), controllable RPO, and architectural flexibility. The main engineering challenge is coordinating three states during failover—determining which messages have reached the backup, which remain in the sync queue, and which are lost—requiring careful state management to avoid duplicate or missing consumption.

Layered Sync Strategy

Production systems rarely rely on a single sync mode; instead they tier messages by business impact, cost, and SLA. For an e‑commerce platform:

Order creation and payment callbacks demand strong sync (zero data loss).

User behavior logs can be async (minor loss acceptable).

Inventory deduction and coupon redemption use semi‑sync (critical but not worth full latency cost).

This tiered approach concentrates ~5 % of critical messages on most of the cross‑datacenter bandwidth, while the remaining 95 % travel cheap async channels.

Choosing the Data Channel

Most mainstream queues implement cluster‑level replication (e.g., Kafka MirrorMaker 2, RocketMQ DLedger Controller, Pulsar Geo‑Replication). While this hides complexity from applications, it also requires synchronizing metadata such as topics, offsets, and message attributes. Offsets often use a lighter async sync but must lag behind message sync to avoid consumers pointing to unsynced data.

Primary‑Standby Failover Engineering

Failover involves four key problems:

Switch Decision: Determining when the primary truly fails; solutions include an independent arbitration service deployed in a separate AZ to achieve multi‑party consensus.

Message Deduplication: Producers may retry to the backup after a switch, generating duplicates; therefore, all cross‑datacenter workloads must be idempotent.

Offset Recovery: After failover, consumers must resume from a consistent offset; most designs prefer a strategy that may duplicate but never lose messages.

Cut‑back: Returning traffic to the restored primary is harder than the initial switch because the primary must catch up; many teams simply keep the former backup as the new primary.

Special Challenges at Ten‑Million QPS

At this scale (≈1 KB per message), cross‑datacenter bandwidth reaches 10 GB/s. A single 100 Gbps link can only sustain ~8 GB/s, so multiple links are required, driving annual bandwidth costs into the tens of millions. Compression, deduplication, and merging become mandatory.

Link stability issues are amplified: a 100 ms jitter translates to millions of messages delayed, demanding multi‑path redundancy and dynamic failover.

Replication delay monitoring must track three dimensions—byte lag, record lag, and time lag—because a “second‑level” delay at ten‑million QPS means millions of unsynced messages.

Hotspot topics (tens of thousands of QPS) need dedicated sync channels to avoid head‑of‑line blocking.

Evolution Path and Future Outlook

The progression follows:

~100 k QPS: pure async suffices.

~1 M QPS: introduce semi‑sync for critical messages.

~10 M QPS: adopt fine‑grained layered sync, optimizing bandwidth per message class.

Future systems may employ "intelligent sync" that dynamically selects a strategy based on message content, business tags, and link health.

Conclusion

Cross‑datacenter sync is a three‑way trade‑off among availability, latency, and cost. No single solution fits all scenarios; the optimal choice depends on the specific business requirements, network conditions, and budget. Moreover, replication alone does not guarantee continuity—idempotent business logic, graceful degradation, and thorough testing remain essential.