Ensuring SpringBoot Message Idempotency to Prevent Duplicate Consumption

Root causes of duplicate consumption

Consumer ACK timeout : Business succeeds but ACK is lost due to network jitter or timeout, so the MQ retries the message.

Consumer abnormal exit : The process crashes before sending ACK, causing the message to be redelivered.

Producer retransmission : Retry mechanisms, interface resends, or network retransmission send identical messages multiple times.

MQ cluster failover : Master‑slave switch, node restart, or partition rebalance leads to the same message being dispatched again.

Message idempotency

Idempotency means that executing the business logic once or N times yields exactly the same final result, without creating dirty, duplicate, or abnormal data. The core goal for MQ is to guarantee that a single message takes effect only once, regardless of how many times it is consumed.

All idempotent solutions rely on a unique message identifier (e.g., msgId, orderId, tradeId, businessId).

Idempotent solutions

Solution 1: Redis SETNX guard

Principle : Use Redis SETNX atomic command to occupy a lock keyed by the global unique msgId. If the lock is acquired, the message is processed; otherwise it is discarded.

Each message carries a globally unique msgId.

Before consumption, attempt to set the key with SETNX.

If the set succeeds, execute business logic.

If the set fails, treat the message as duplicate and ACK it immediately.

Set an expiration time to avoid stale keys.

Code

import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import javax.annotation.Resource;
import java.util.concurrent.TimeUnit;

@Component
public class MqIdempotentRedisUtil {
    @Resource
    private StringRedisTemplate stringRedisTemplate;

    /**
     * Message idempotent lock
     * @param msgId unique message ID
     * @param expireSeconds expiration time (greater than max business execution time)
     * @return true = first consumption, false = duplicate
     */
    public boolean tryLock(String msgId, long expireSeconds) {
        String key = "mq:idempotent:" + msgId;
        // SETNX atomic operation: set if absent, return false if exists
        return stringRedisTemplate.opsForValue()
                .setIfAbsent(key, "consumed", expireSeconds, TimeUnit.SECONDS);
    }
}

Pros : High performance, no DB pressure, works with any MQ, simple code, non‑intrusive.

Cons : Relies on Redis; if Redis fails, a fallback strategy is needed.

Applicable scenarios : Most internet services, low‑to‑medium concurrency cases.

Solution 2: Database unique index guard

Principle : Create a dedicated table with a unique index on msgId. Inserting a record before processing serves as the idempotent check.

Insert a guard record with the message ID.

If insertion succeeds, this is the first consumption; proceed with business logic.

If a unique‑key violation occurs, the message is duplicate and is discarded.

Table definition

CREATE TABLE mq_message_record (
    id BIGINT AUTO_INCREMENT PRIMARY KEY COMMENT 'Primary key',
    msg_id VARCHAR(64) NOT NULL COMMENT 'Unique message ID',
    business_type VARCHAR(32) COMMENT 'Business type',
    create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
    UNIQUE INDEX uk_msg_id (msg_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT 'MQ message deduplication table';

Code

import org.springframework.dao.DuplicateKeyException;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import javax.annotation.Resource;

@Service
public class MqIdempotentDbService {
    @Resource
    private MqMessageRecordMapper messageRecordMapper;

    @Transactional(rollbackFor = Exception.class)
    public boolean isFirstConsume(String msgId, String businessType) {
        try {
            MqMessageRecord record = new MqMessageRecord();
            record.setMsgId(msgId);
            record.setBusinessType(businessType);
            messageRecordMapper.insert(record);
            return true;
        } catch (DuplicateKeyException e) {
            // Duplicate key means duplicate message
            return false;
        }
    }
}

Pros : No middleware dependency, strong transactional consistency, absolutely reliable, can serve as a Redis fallback.

Cons : High DB pressure under heavy concurrency, slower than Redis.

Applicable scenarios : Core finance, payment, accounting, or when Redis is unavailable.

Solution 3: Business state machine + optimistic lock

Principle : For workflows with clear state transitions (e.g., order → payment → shipment → completion), allow only forward state changes. Once a state is changed, further updates to that state are rejected, achieving natural idempotency.

State flow example: Pending Payment (1) → Paid (2) → Shipped (3) → Completed (4) .

SQL example

UPDATE order_info
SET status = 2, pay_time = NOW()
WHERE order_id = #{orderId} AND status = 1;

Java example

@Service
public class OrderService {
    @Resource
    private OrderMapper orderMapper;

    @Transactional(rollbackFor = Exception.class)
    public boolean paySuccess(Long orderId) {
        // rows == 0 means the order was already processed → duplicate consumption
        int rows = orderMapper.updateOrderStatus(orderId, 1, 2);
        return rows > 0;
    }
}

Pros : Zero extra storage, zero overhead, highest business fit, absolute idempotency.

Cons : Only works for stateful business; stateless scenarios cannot use it.

Applicable scenarios : Orders, payments, refunds, points changes, membership rights updates.

Solution 4: Global unique constraint

Some domains already have a natural unique key (e.g., payment transaction number, order ID, refund number). Inserting data can directly check the primary key; if it exists, the operation is skipped.

Payment transaction number is unique.

Order ID is unique.

Refund number is unique.

This approach suits simple “create‑only” message scenarios.

Precautions

Lock expiration must exceed business execution time . If the lock expires early, duplicate messages may slip through. Recommended: lock timeout = 3 × max business duration.

Idempotent check before business execution . Correct order: idempotent check → business execution → manual ACK.

Never use automatic ACK . Automatic ACK confirms the message before the business finishes, preventing retries and breaking idempotency. All core‑business MQs should use manual ACK.

msgId must be globally unique and non‑empty . Producers should generate IDs using UUID, Snowflake, etc.

Do not delete Redis lock immediately after execution . Deleting too early can cause a race where another consumer acquires the lock before it expires. Rely on the expiration time instead.

Summary

Duplicate consumption is inevitable; root causes are ACK timeout, consumer crash, cluster failover, and producer retransmission.

Message idempotency hinges on a unique message ID combined with a pre‑consumption deduplication check.

The generally optimal solution is Redis SETNX guard, suitable for all MQ scenarios.

Fallback or complementary solutions include database unique index, state‑machine with optimistic lock, and natural unique constraints.

Production best practices: manual ACK, appropriate lock timeout, pre‑emptive deduplication, and globally unique msgId.