Handling Duplicate Messages, Ordering, Concurrency, and Batch Processing in Message‑Driven Systems

Yuzhaohui, who joined Qunar's technology team in 2011, introduces his experience building reliable messaging middleware, task scheduling, and order services, emphasizing performance optimization, parallelism, and clean code.

Qunar has used a message‑driven architecture for over four years, relying on messages to drive the entire transaction chain. The article summarizes the patterns they have accumulated and uses QMQ as a concrete example.

1. Consumer handling of duplicate messages

Message consumption can follow three delivery semantics: at‑most‑once, at‑least‑once, and exactly‑once. The most reliable is exactly‑once, but it often requires additional mechanisms such as idempotent checks.

1.2 Duplicate‑message handling strategies

1.2.1 No handling : In scenarios where occasional duplicate cache‑eviction messages have negligible impact, ignoring duplicates is acceptable.

1.2.2 Business handling : Design business logic to be idempotent; for example, treat a duplicate order creation as a successful response without persisting a second record.

1.2.3 Deduplication table : Store a unique key extracted from the message in a deduplication table (Redis or a database). On receipt, check the table and skip processing if the entry already exists. QMQ provides a built‑in idempotent checker that can be configured easily.

When a message updates a database and then calls another service, a failure after the DB update can cause a retry and potential duplicate service calls. Therefore, services should be designed to be idempotent.

2. Message ordering

2.1 No handling : For non‑critical ordering, the same deduplication approaches used for duplicates can be applied.

2.2 Business handling : Most business logic can tolerate out‑of‑order messages; for state‑machine transitions, simply reject out‑of‑order messages and let later messages correct the state.

2.3 Extra fields : Non‑strict ordering: Use an updated_time timestamp to ignore stale messages. Strict ordering: Add a version field and apply optimistic‑lock updates (e.g., UPDATE tbl SET ..., version=version+1 WHERE [email protected] ). If the update fails, throw an exception to trigger a retry.

3. Concurrent updates

Version fields are also used to control concurrent updates when merging JSON payloads stored in a database. The typical pattern is to read‑modify‑write with version checking, retrying on conflict. QMQ provides a NeedRetryException to simplify this.

4. Asynchronous processing

When message handling spawns asynchronous work, explicit acknowledgments are required. Developers must call ack() after the async task completes; misuse (e.g., creating extra thread pools or forgetting to ack) leads to message loss or unnecessary retries.

5. Batch processing

5.1 Batch when possible : If consumption speed exceeds production, batching may be unnecessary; otherwise, batch messages to improve throughput. QMQ offers a BatchExecutor class.

5.2 Time‑or‑size batching : Process a batch when either a maximum size or a timeout is reached, whichever comes first. QMQ includes a ready‑to‑use implementation.

5.3 Multi‑channel batch : For sharded databases, group messages by target shard and batch each group separately. QMQ provides a pattern for this scenario.

The article concludes that these patterns, combined with QMQ's built‑in features, help build robust, high‑performance message‑driven systems.