How to Achieve Exactly‑Once Message Processing in RocketMQ Without Transactions

Message Middleware Reliability and At‑Least‑Once Semantics

Message middleware guarantees that once a message is successfully stored in the broker, it will be delivered to a consumer at least once. If a consumer crashes before acknowledging the message, the broker will redeliver the same message until the consumer acknowledges successful processing.

Simple Deduplication Approach

Typical consumption logic inserts an order record and updates inventory:

insert into t_order values ...;
update t_inv set count = count-1 where good_id = 'good123';

A naïve idempotence check queries the order table before insertion:

select * from t_order where order_no = 'order123';
if (order != null) {
    return; // duplicate, skip
}

This works when messages are processed sequentially but fails under high concurrency because the second message may read the table before the first transaction commits.

Concurrent Duplicate Issue

When two messages arrive within the processing window, the second may see an empty order record, insert a duplicate, cause primary‑key conflicts, or deduct inventory twice.

Concurrency Solution: SELECT FOR UPDATE

Wrap the check in a transaction and lock the row:

select * from t_order where order_no = 'THIS_ORDER_NO' for update;
if (order.status != null) {
    return; // duplicate, skip
}

This prevents concurrent inserts but increases transaction duration and reduces throughput.

Exactly‑Once Semantics

Exactly‑Once means a message is processed by the consumer only once, even if the producer retries sending.

Achieving true exactly‑once without relying on relational‑database transactions is difficult.

Transactional Approach Using a Message Table

Insert a record into a dedicated message_log table within the same transaction that updates the business table:

BEGIN;
INSERT INTO message_log (msg_id, status) VALUES (...);
UPDATE t_order SET status='SUCCESS' WHERE order_no='order123';
COMMIT;

Steps:

Start transaction

Insert message record (handle primary‑key conflicts)

Update business table

Commit transaction

If the transaction commits, the message record guarantees that the message will not be reprocessed even if the broker redelivers it. If the transaction aborts, the broker will retry because the consumption offset is not advanced.

Limitations:

Only works when the entire consumption logic can be wrapped in a single relational‑database transaction.

Cannot cover non‑transactional stores such as Redis.

Cross‑database transactions are not supported.

Complex Business Scenarios

In multi‑step workflows (e.g., inventory check, lock, order insert, downstream RPC calls, status update, commit), many steps cannot be included in a single DB transaction, making the message‑table approach insufficient.

Decomposing Message Processing

One strategy is to split the workflow into several sub‑messages, each handled by its own transaction. This reduces the scope of each transaction but adds latency and architectural complexity.

General Non‑Transactional Deduplication Using a Message Status Table

Maintain a message table with a status field (e.g., processing, completed). Only messages marked as completed are considered idempotent. Concurrent duplicates are routed to a retry topic until the first processing finishes.

To avoid indefinite retries, attach an expiration time (e.g., 10 minutes) to the processing state; if the timeout expires, the record is removed so the message can be retried.

Using Redis as a Flexible Storage Medium

Redis can serve as the message‑status store, offering lower latency and built‑in TTL for expiration. This sacrifices the strong consistency guarantees of MySQL but simplifies cleanup of stale processing entries.

Open‑Source Implementation

The pattern is implemented in the RocketMQDedupListener project on GitHub: https://github.com/Jaskey/RocketMQDedupListener

Example using Redis for deduplication:

// Use Redis for deduplication
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1");
consumer.subscribe("TEST-TOPIC", "*");
String appName = consumer.getConsumerGroup(); // usually the consumer group name
StringRedisTemplate stringRedisTemplate = null; // obtain from Spring context
DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);
DedupConcurrentListener listener = new SampleListener(dedupConfig);
consumer.registerMessageListener(listener);
consumer.start();

The only modification to standard RocketMQ code is the creation of a DedupConcurrentListener that handles the deduplication key (default is messageId).

Value and Limitations

This approach solves the majority of duplicate‑message problems caused by broker retries or producer resends and controls concurrent duplicate consumption. However, it does not guarantee idempotence when intermediate steps (e.g., inventory lock) are not themselves idempotent or lack rollback mechanisms.

Best‑Practice Recommendations

Ensure consumer code can roll back on failure.

Implement graceful shutdown to avoid mid‑process crashes.

Make non‑idempotent operations abort with alerts.

Monitor consumption failures and intervene manually when necessary.

Non‑Transactional Deduplication Flow (Diagram 1)

The flow uses a status column to distinguish processing and completed states. Duplicate messages arriving while a record is in processing are sent to a retry topic, preventing premature offset advancement.

Handling Stale Processing State (Diagram 2)

If a message remains in processing longer than the configured timeout (e.g., 10 minutes), the record is removed so the message can be retried, avoiding dead‑letter accumulation.