Zero Message Loss at 10M QPS: From Best‑Effort to Proven Guarantees

一次集体怀疑人生的“消息蒸发”

During a pre‑Double‑11 load test, the production side logged 10 million order events while the consumer side only processed about 9.98 million, with the missing ~20 k messages never appearing in broker inbound metrics despite successful send callbacks. The root cause was asynchronous sends with callbacks that merely logged results, lacking retries or persistence; a brief broker node hiccup caused thousands of in‑flight messages to be silently dropped from the client’s local buffer.

An internal three‑year incident review showed that only 20% of message loss stemmed from actual broker hardware failures, while the remaining 80% originated from “theoretically impossible” loss points across the three link stages: production callbacks, insufficient broker replica counts, premature consumer offset commits, and in‑flight loss during rebalance.

消息生命周期里的三个高发地带

The message journey comprises three distinct links, each with its own loss patterns:

Production side : asynchronous send callbacks not handling failures, local buffer clears, send thread crashes, OOM kills—these failures are silent to business code.

Broker side : single‑replica writes, page‑cache‑only flushes, ISR shrink‑induced leader loss, disk corruption, accidental topic deletion—easily monitored but hard to remediate.

Consumer side : offset commits before processing, batch‑level ack despite partial failures, in‑flight loss during rebalance, out‑of‑order commits.

Any system claiming “no message loss” must provide verifiable guarantees in all three links; fixing only one segment will inevitably be exposed by the others.

生产侧：从 fire-and-forget 到 ack 闭环

Typical loss occurs when developers treat asynchronous send as synchronous; the SDK queues the message locally, and if the downstream chain fails, the message vanishes without error logs or alerts.

To upgrade from best‑effort to guaranteed delivery, a full ack loop is required. At ten‑million QPS, many teams adopt an outbox table: the business transaction writes the message with a “pending” status, and a background task scans the table to send messages to the broker, updating the status upon success. This shifts the reliability dependency from network/broker availability to local database durability.

Kafka’s acks=all and RocketMQ’s synchronous double‑write provide broker‑side replica guarantees, but they do not replace producer‑side ack handling; without processing the ack callback or retrying after network interruptions, broker guarantees are moot.

Broker 侧：副本、刷盘与持久化语义

Broker reliability involves multiple coupled switches beyond a simple “replication factor 3” setting. Key dimensions include:

Sync vs async flush : sync flush writes each message to disk inode (throughput drops by an order of magnitude); async flush relies on OS page cache and background fsync, risking loss if the OS crashes within milliseconds.

Sync vs async replica : sync replica requires all ISR members to write successfully before ack, preventing loss on single‑node failure; async replica returns after leader write, leaving a data hole if the leader crashes before followers catch up.

ISR count : when ISR shrinks to one, sync replica degrades to single‑replica. The min.insync.replicas setting protects against this.

At ten‑million QPS, “sync flush + full‑sync replica” is generally infeasible due to latency pressure. A common practice is tiered reliability: critical financial topics use sync flush and full replication, while less critical log or telemetry topics use async flush with partial replication.

Note that OS‑level fsync does not guarantee hardware persistence; RAID cache, SSD power‑loss caches, and HDD write buffers require battery‑backed units or NVMe with power‑loss protection for true durability.

消费侧：处理失败也是丢消息的一种

Beyond offset‑early‑commit issues discussed earlier, a frequent loss pattern is swallowing exceptions in consumer code (e.g., try‑catch that logs and continues). The message is considered processed by the broker, offset advances, yet the business state never changes.

To avoid this silent loss, consumers must classify each message into one of four explicit outcomes (success, retryable failure, non‑retryable failure, dead‑letter). Messages that repeatedly fail should be routed to a dead‑letter queue, which must be paired with monitoring and SOPs to prevent it from becoming a hidden trash bin.

端到端的“可观测丢失”

Proving that no messages are lost requires the three counts—produced, broker‑received, and consumed—to match. A typical reconciliation pipeline at ten‑million QPS includes:

Producer metrics : per‑topic per‑minute counters of successful sends.

Broker metrics : inbound rate counters per topic.

Consumer metrics : per‑topic per‑minute counters of successful processing.

Reconciliation task : minute‑level comparison of the three numbers, triggering alerts when thresholds are exceeded.

Adding trace IDs to critical messages enables pinpointing the exact link where loss occurs, at the cost of extra bandwidth and storage.

兜底防线：补偿、回放、重放

Even with all safeguards, zero loss is an asymptotic goal; rare failures still happen. The final safety net consists of compensation and replay mechanisms:

Production side : the outbox table serves as a natural replay source for unsent messages.

Broker side : rely on the business system as the truth source to republish lost messages; the message system should never be the sole data store.

Consumer side : use broker‑provided time‑ or offset‑based replay, ensuring idempotent processing to avoid new failures.

The essence of reliability engineering is not “no problems” but “quickly detect, recover, and prove recovery” after a fault.

千万 QPS 下的丢失阈值化

Absolute zero loss at ten‑million QPS would require synchronous flush, full‑sync replication, transactional producer messages, transactional consumer commits, and real‑time reconciliation—reducing single‑node throughput to one‑tenth and inflating hardware costs dramatically. The cost often outweighs the occasional loss impact.

Mature practice is to SLA‑grade loss rates per topic, defining concrete configuration bundles for each tier. Low‑SLA topics tolerate minor deviations without incident classification, while high‑SLA topics trigger post‑mortems for any deviation, focusing human effort on the most critical paths.

写在结尾：从尽力到可证明

The evolution from “best‑effort” to “provable, recoverable” reliability passes three mindsets: initially trusting the framework, then adding layered defenses (acks, replicas, manual commits), and finally instituting counting, reconciliation, compensation, and SLA enforcement. The true meaning of “no message loss” is not “zero drops” but “instant detection and guaranteed recovery when a drop occurs”.