Message Middleware: Benefits, Drawbacks, and Design Patterns for Concurrency, Ordering, Duplicate, and Transactional Messaging

Overview

In microservice development, message middleware is often introduced to achieve business decoupling and asynchronous processing. The three core benefits are decoupling, asynchrony, and peak‑shaving, while the main drawbacks include potential performance bottlenecks, single‑point‑of‑failure risks, and added operational complexity.

Handling Concurrency and Ordered Messages

When multiple consumer instances are deployed to increase throughput, the challenge is to ensure each message is processed exactly once and in the order it was sent. For example, three identical consumers reading from the same channel may receive Order Created, Order Updated, and Order Cancelled out of order, causing inconsistent behavior.

Kafka solves this by using partitions: a topic is split into multiple partitions, the producer assigns a partition key (e.g., orderId), and consumers belonging to the same consumer group are assigned partitions such that each partition is consumed by only one instance, preserving order.

By using the same orderId as the partition key, all events of a specific order are routed to the same partition and thus processed by the same consumer instance in order.

Handling Duplicate Messages

Message brokers typically guarantee at‑least‑once delivery, which can lead to duplicate messages when failures occur. Two main strategies exist: writing idempotent message handlers or tracking message IDs and discarding duplicates.

Idempotent Message Handlers

If the business logic is idempotent (e.g., cancelling an already‑cancelled order), duplicate messages are harmless.

Tracking and Discarding Duplicates

A common approach is to store the messageId of each processed message in a database table (e.g., PROCESSED_MESSAGE) and ignore any incoming message whose ID already exists.

Handling Transactional Messages

Services often need to publish messages as part of a database transaction; otherwise, a crash after the DB update but before the message send can leave the system inconsistent.

Transactional Outbox Pattern

When using a relational database, an outbox table stores messages within the same ACID transaction as the business data. A separate message‑relay service reads pending rows from the outbox and publishes them to the broker, then deletes the rows.

Change Data Capture (CDC) via Transaction Log

Another approach uses the database's transaction log (e.g., MySQL binlog) to capture changes. Tools like Alibaba's canal simulate a MySQL slave, read the binary log, and forward parsed events to the message broker.

More details: Alibaba Canal GitHub

RocketMQ Transactional Message Solution

RocketMQ (since version 4.3.0) supports distributed transactional messages using a two‑phase commit (2PC) model with compensation logic for time‑outs. The flow includes sending a half‑message, executing the local transaction, and then committing or rolling back based on the transaction outcome. If the broker does not receive the transaction status, it triggers a check‑back request to the producer.

In practice, developers create a transactional log table and annotate the method with @Transactional so that both the business operation and the log insertion are atomic. If a log entry exists, the transaction is considered successful and the message is committed; otherwise, it is rolled back.

For a deeper dive, see the article RocketMQ Advanced – Transactional Messages .