0 views collected around this technical thread.
This article explains the three message delivery semantics, the common causes of duplicate messages in queue systems, and presents concrete producer‑side, broker‑side, and consumer‑side deduplication techniques for Kafka and Pulsar, including code samples and best‑practice recommendations.
Message queues can deliver duplicates under at‑least‑once semantics, so to protect idempotent business logic such as orders and payments you should combine producer‑side idempotence (e.g., Kafka’s enable.idempotence), broker‑side deduplication (e.g., Pulsar), and a consumer‑side guard using unique IDs stored in a database or Redis.
This article shares Qunar's practical experiences with message-driven architecture, detailing consumer handling of duplicate messages, ordering, concurrency control, asynchronous processing, and batch strategies, and presents concrete solutions such as idempotent checks, deduplication tables, versioning, and QMQ's built‑in executors.