This article breaks down Kafka’s message delivery guarantees—at‑most‑once, at‑least‑once, and exactly‑once—explaining the underlying producer, broker, and consumer requirements, configuration settings, and practical code snippets to achieve reliable delivery.
Tencent’s QQ reminder subscription push system combines push‑pull prefetching, heterogeneous MySQL/Redis storage, delayed‑queue triggers, Redis‑based rate‑limiting, task scattering, message‑queue decoupling, at‑least‑once delivery, and dual‑replica disaster recovery within a dispatcher‑worker architecture to ensure reliable, controllable, low‑latency mass notifications.
This article explains Kafka's storage architecture—including topics, partitions, segments, and their naming rules—along with how data is read, and details the system's reliability features such as ISR/OSR replication, leader election, producer acknowledgment levels, and delivery guarantees.
YouZan’s experience with Flume shows how the at‑least‑once delivery model, combined with FileChannel storage and custom extensions such as an NsqSource, hourly‑based HdfsEventSink, metric reporting server, and timestamp interceptor, can reliably move MySQL binlog data to HDFS, while tuning transaction batch size and channel capacity boosts throughput and stability, paving the way for a unified management platform.
This article explains how network unreliability makes message delivery in distributed systems complex, covering network request outcomes, timeout handling, the three delivery semantics—at‑most‑once, at‑least‑once, exactly‑once—and how protocols like AMQP and MQTT address these challenges.
The article explains how messages travel in monolithic and distributed systems, why network communication is inherently unreliable, and how different delivery semantics—at‑most‑once, at‑least‑once, and exactly‑once—affect reliability, ordering, and protocol choices such as AMQP and MQTT.
The article explains how unreliable network communication in distributed systems leads to three message delivery semantics—at‑most‑once, at‑least‑once, and exactly‑once—detailing their mechanisms, ordering challenges, and common protocols such as AMQP and MQTT.
