How to Achieve Exactly‑Once Message Processing in RocketMQ Without Transactions

Message Middleware Reliability

Message middleware is a common component in distributed systems, providing asynchronous decoupling, peak‑shaving, and other benefits. Its basic reliability guarantee is "at‑least‑once" delivery, meaning that once a message is successfully sent to the middleware it will be consumed at least once.

When a consumer crashes after processing a message but before acknowledging it, the middleware will redeliver the same message, leading to duplicate consumption. In RocketMQ this appears as repeated delivery of the same messageId.

Simple Deduplication Approach

A typical way to achieve idempotence is to check whether the business record already exists before processing:

insert into t_order values ...;
update t_inv set count = count-1 where good_id = 'good123';

And before inserting:

select * from t_order where order_no = 'order123';
if (order != null) {
    return; // duplicate, skip
}

This works in many cases but fails under high concurrency because two consumers may see the record as absent simultaneously.

Concurrent Duplicate Messages

When duplicate messages arrive within the processing window (e.g., 100 ms), the simple check may miss the race condition, causing duplicate business logic execution.

One Concurrency‑Safe Solution

Lock the row during the check using SELECT ... FOR UPDATE inside a transaction:

select * from t_order where order_no = 'THIS_ORDER_NO' for update;
if (order.status != null) {
    return; // duplicate, skip
}

This ensures only one consumer proceeds, but it prolongs the transaction and reduces throughput.

Exactly‑Once Semantics

Exactly‑once means a message is processed successfully exactly one time, even if the producer retries sending it. Alibaba Cloud defines it as the consumer handling the message only once, regardless of producer retries.

In practice, achieving exactly‑once is difficult without relying on database transactions.

Transaction‑Based Exactly‑Once Using a Message Table

One approach is to create a dedicated message‑consumption table and perform the following steps within a single database transaction:

Begin transaction

Insert a record into the message table (handling primary‑key conflicts)

Update the business table (e.g., order status)

Commit transaction

If the transaction commits, the message record is persisted, guaranteeing that even if the broker retries delivery the insert will fail and the message is considered already processed. If the service crashes before commit, both the business update and the message record are rolled back, so the broker will retry and the message will be processed again.

This method underlies Alibaba Cloud ONS’s exactly‑once implementation.

Limitations of the Transaction Approach

It requires the entire consumption logic to be wrapped in a relational‑database transaction.

Non‑transactional resources (e.g., Redis, external RPC calls) cannot be rolled back.

Cross‑database scenarios are unsupported.

Therefore, it is not universally applicable.

Non‑Transactional Deduplication with an Idempotent Table

Instead of a transaction, we can maintain a dedicated idempotent table with a status column (e.g., processing, completed). Only messages marked completed are considered processed; duplicates in processing trigger delayed retries.

The flow (illustrated below) avoids transactions and relies on the uniqueness of the business key (e.g., order number) as the primary key.

To prevent indefinite “processing” states, a TTL (e.g., 10 minutes) can be set so that stale entries are cleared, allowing retries.

Using Redis as the Idempotent Store

Redis can serve as the idempotent table, offering lower latency and built‑in TTL support. However, it sacrifices the strong consistency guarantees of relational databases.

Sample Java Listener with Redis Deduplication

// Use Redis for the idempotent table
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1");
consumer.subscribe("TEST-TOPIC", "*");
String appName = consumer.getConsumerGroup();
StringRedisTemplate stringRedisTemplate = null; // obtain from Spring context
DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);
DedupConcurrentListener messageListener = new SampleListener(dedupConfig);
consumer.registerMessageListener(messageListener);
consumer.start();

The only change to the original RocketMQ consumer code is the creation of a DedupConcurrentListener that handles deduplication based on a configurable business key (default is messageId).

When Is This Solution Sufficient?

For the vast majority of cases—where failures are rare—the non‑transactional deduplication solves 99 % of duplicate‑message problems caused by broker retries or producer re‑sends. To handle the remaining edge cases, consider:

Ensuring consumer code can roll back on failure.

Implementing graceful shutdown to avoid mid‑process crashes.

Making non‑idempotent operations (e.g., stock lock) fail loudly if repeated.

Monitoring retry patterns and manually intervening when necessary.

In summary, while no method guarantees perfect idempotence in every failure scenario, the described approaches provide practical, reusable tools for achieving reliable exactly‑once processing in RocketMQ.