MQ Duplicate Consumption: Three Idempotency Patterns with Code & Pitfalls
This article explains why MQ duplicate consumption occurs due to ACK loss, and details three idempotency solutions—database unique index, Redis atomic operations, and business state machines—with code examples, comparison tables, production pitfalls like race conditions and transaction boundaries, best practices, and interview-focused implementation insights.
Root Cause of Duplicate Consumption
The fundamental cause is the conflict between message acknowledgment mechanisms and network unreliability: the consumer finishes processing a message, but the ACK fails to reach the MQ server, causing the broker to treat the message as failed and trigger a redelivery.
General Solution: Idempotency Design
The only correct approach is to make the consumption operation itself idempotent—processing the same message multiple times yields the same result as processing it once.
Solution 1: Database Unique Index (Simplest & Most Efficient)
Technical highlight: Leverages database atomicity; no extra query needed; optimal performance.
@Service
@Slf4j
public class OrderPayConsumer {
@Autowired
private DuplicateRecordMapper duplicateRecordMapper;
@Autowired
private OrderService orderService;
@Transactional(rollbackFor = Exception.class)
public void onMessage(MessageExt message) {
// Extract business unique identifier (prefer business ID over MQ messageId)
String orderId = JSON.parseObject(message.getBody(), OrderMessage.class).getOrderId();
try {
// Insert deduplication record first; unique index conflict throws exception
// Technical highlight: uses database atomicity to avoid "check-then-insert" race condition
duplicateRecordMapper.insertSelective(
DuplicateRecord.builder()
.businessId(orderId)
.businessType("ORDER_PAY")
.messageId(message.getMsgId())
.createTime(new Date())
.build()
);
// Execute business logic
orderService.updateOrderStatusToPaid(orderId);
log.info("Order payment message processed successfully, orderId: {}", orderId);
} catch (DuplicateKeyException e) {
// Duplicate message, return success directly (tell MQ not to retry)
log.warn("Duplicate message detected, ACK directly, orderId: {}, messageId: {}", orderId, message.getMsgId());
} catch (Exception e) {
// Business exception, throw to let MQ retry
log.error("Order payment message processing failed, orderId: {}", orderId, e);
throw new RuntimeException("Message processing failed", e);
}
}
}Solution 2: Redis Distributed Lock (High-Concurrency Scenarios)
Technical highlight: Uses Lua script to guarantee atomicity, preventing accidental lock deletion.
@Component
@Slf4j
public class RedisIdempotentUtil {
@Autowired
private StringRedisTemplate redisTemplate;
// Lua script: atomically check lock existence and delete
private static final String UNLOCK_SCRIPT =
"if redis.call('get', KEYS[1]) == ARGV[1] then " +
"return redis.call('del', KEYS[1]) " +
"else return 0 end";
/**
* Try to acquire idempotent lock
* @param businessId business unique identifier
* @param expireTime expiration time (seconds)
* @return whether acquisition succeeded
*/
public boolean tryLock(String businessId, long expireTime) {
String key = "idempotent:order_pay:" + businessId;
// Use UUID as lock value to prevent deleting another thread's lock
String value = UUID.randomUUID().toString();
Boolean result = redisTemplate.opsForValue()
.setIfAbsent(key, value, Duration.ofSeconds(expireTime));
return Boolean.TRUE.equals(result);
}
/** Release idempotent lock */
public void unlock(String businessId, String value) {
String key = "idempotent:order_pay:" + businessId;
redisTemplate.execute(
new DefaultRedisScript<>(UNLOCK_SCRIPT, Long.class),
Collections.singletonList(key), value);
}
}Solution 3: Business State Machine (Most Elegant)
Technical highlight: Uses database row locks to guarantee atomic state updates.
@Mapper
public interface OrderMapper {
/**
* Atomically update order status
* Technical highlight: WHERE clause ensures only pending-payment status can update to paid
* Simultaneously uses database row lock to prevent concurrent updates
*/
@Update("UPDATE orders " +
"SET status = #{newStatus}, update_time = NOW() " +
"WHERE id = #{orderId} AND status = #{oldStatus}")
int updateStatus(@Param("orderId") String orderId,
@Param("oldStatus") String oldStatus,
@Param("newStatus") String newStatus);
}
// Business layer call
@Transactional
public void updateOrderStatusToPaid(String orderId) {
int affectedRows = orderMapper.updateStatus(orderId, "UNPAID", "PAID");
if (affectedRows == 0) {
// Order status is not pending payment, already processed
log.warn("Order status already updated, no need to reprocess, orderId: {}", orderId);
return;
}
// Execute subsequent business logic (deduct inventory, create logistics order, etc.)
inventoryService.deductStock(orderId);
logisticsService.createLogisticsOrder(orderId);
}Comparison of Solutions
Database Unique Index – Implementation difficulty: ⭐, Performance: ⭐⭐⭐, Applicable: simple insert operations, Drawback: depends on database, potential performance bottleneck.
Redis Distributed Lock – Difficulty: ⭐⭐, Performance: ⭐⭐⭐⭐, Applicable: high-concurrency complex business, Drawback: complex implementation, must handle lock expiration.
Message Deduplication Table – Difficulty: ⭐⭐, Performance: ⭐⭐, Applicable: general scenarios, Drawback: increases database pressure.
Business State Machine – Difficulty: ⭐⭐⭐, Performance: ⭐⭐⭐⭐⭐, Applicable: stateful business flows, Drawback: tightly coupled to business.
Core Technical Challenges and Solutions
"Check-Then-Insert" Race Condition – Two threads simultaneously query and find message unconsumed, both execute business logic. Solution: use database unique index, Redis atomic operations; forbid check-then-insert.
Distributed Lock Expiration – Business processing exceeds lock TTL, lock auto-releases. Solution: set reasonable expiration (3-5x business duration), use watchdog auto-renewal, add secondary validation in business logic.
Dead Letter Queue Handling – Messages failing multiple retries enter DLQ unattended. Solution: configure DLQ alerts, establish manual processing workflow, periodically clean expired DLQ messages.
Deduplication Table Bloat – Accumulating records degrade query performance. Solution: time-based partitioning, periodic archiving, set data TTL (e.g., retain 30 days).
Message Out-of-Order – Redelivered message may arrive before subsequent messages. Solution: business state machine naturally supports out-of-order; partition consumption by business ID; add version control.
Transaction Consistency – Business logic succeeds but deduplication record insertion fails. Solution: use local transaction for atomicity; insert deduplication record first, then execute business; rollback entire transaction on exception.
Best Practices
Prefer business unique identifier over MQ-generated messageId.
Consumer performs idempotency check first, then executes business logic.
Set reasonable retry count (typically 3-5), then route to dead letter queue.
Periodically clean expired deduplication records to avoid table bloat.
Monitor duplicate consumption count to detect anomalies early.
All MQ consumers must implement idempotency—fundamental requirement for distributed systems.
Interview Simulation
Interviewer: "How to solve MQ duplicate consumption?"
Candidate explains three causes: producer timeout without ACK causing resend; consumer crash before offset commit leading to re-fetch; Kafka rebalance causing overlapping consumption. Summary: MQ guarantees at-least-once delivery, not exactly-once; idempotency must be handled by business side.
Scenario: Order payment callback, same message may be called twice.
Candidate chooses database deduplication table. Core idea: message contains business serial number (e.g., outTradeNo). Create payment_dedup table with unique index on that field. On consumption, attempt insert; if success, process business (update order status) within same local transaction; if unique key conflict, treat as duplicate and ACK directly.
Transaction boundary question.
If deduplication insert and business are in separate transactions, a failure after insert but before business commit leaves a stray mark, causing genuine retry to be misjudged as duplicate. Must use same database transaction: insert deduplication + update order status, all-or-nothing.
High QPS optimization.
Introduce Redis lightweight idempotency: SET key value NX EX 1800 (atomic). Key is outTradeNo. If set succeeds, execute business; if exists, skip. Do not delete key after business success; rely on TTL expiration to cover duplicate arrival window. Note: Redis may lose marks during master-slave failover, so financial core paths keep database deduplication as final safety net.
State machine for status updates.
Use version number + affected rows: UPDATE orders SET status='PAID', version=version+1 WHERE id=#{id} AND version=#{oldVersion}. First message updates successfully (rows=1); duplicate with old version yields rows=0, treated as duplicate. Better than WHERE status='WAIT_PAY' because it prevents concurrent overwrites and is naturally idempotent.
Candidate's three typical pitfalls.
Delete cache before business execution; duplicate consumption finds cache missing and re-executes.
Deduplication mark placed outside business transaction, causing misjudgment.
Using MQ's messageId as idempotency key; MQ resend may generate new messageId, breaking idempotency.
Therefore, always use business unique identifier for idempotency key; mark and business must be atomically persisted.
Core Code Examples with Technical Highlights
1. Database Deduplication Table + Business Transaction Atomicity
@Service
public class PaymentCallbackService {
@Autowired
private OrderDao orderDao;
@Autowired
private DedupDao dedupDao;
/**
* Handle payment callback with idempotency guarantee
* @param callbackMsg contains outTradeNo, status, etc.
*/
@Transactional(rollbackFor = Exception.class)
// Highlight 1: overall transaction
public void handlePaymentCallback(CallbackMsg callbackMsg) {
String outTradeNo = callbackMsg.getOutTradeNo();
// Highlight 2: deduplication table insert, unique index auto-detects duplicate
int inserted = dedupDao.insertIfAbsent(outTradeNo);
if (inserted == 0) {
// Duplicate message, return directly, idempotent skip
logger.info("Duplicate message, outTradeNo={}", outTradeNo);
return;
}
// First-time processing, execute business
int affected = orderDao.updateOrderStatus(
outTradeNo,
OrderStatus.PAID.getCode(),
OrderStatus.WAIT_PAY.getCode()
);
if (affected != 1) {
throw new BizException("Order status abnormal, cannot update");
}
}
}
// DedupDao.java
@Mapper
public interface DedupDao {
// INSERT IGNORE or ON DUPLICATE KEY ensures idempotency
@Insert("INSERT IGNORE INTO payment_dedup(out_trade_no, create_time) VALUES(#{outTradeNo}, NOW())")
int insertIfAbsent(@Param("outTradeNo") String outTradeNo);
}Technical Highlights Analysis
Deduplication table insert and business update in same @Transactional – either both succeed or both rollback, preventing stray marks.
Uses INSERT IGNORE instead of SELECT + INSERT, avoiding concurrency race; database unique index guarantees atomicity.
Return value judgment: 0 indicates unique key conflict (duplicate message), business does no processing.
2. Redis Distributed Idempotency (High-Concurrency Version)
@Component
public class RedisDedupService {
@Autowired
private StringRedisTemplate redisTemplate;
// Highlight 1: SET key value NX EX atomic operation
public boolean tryDedup(String bizId, int expireSeconds) {
String key = "msg:dedup:" + bizId;
Boolean result = redisTemplate.opsForValue()
.setIfAbsent(key, "1", Duration.ofSeconds(expireSeconds));
return Boolean.TRUE.equals(result);
}
// Highlight 2: do not actively delete after business completion, rely on TTL expiration
// Prevents deletion earlier than duplicate message arrival window
}
// Usage example
@Service
public class OrderMessageListener {
@Autowired
private RedisDedupService dedupService;
public void onMessage(OrderEvent event) {
String bizId = event.getOrderId() + event.getEventType();
// Deduplication window 30 minutes (greater than message retry cycle)
if (!dedupService.tryDedup(bizId, 30 * 60)) {
return; // Duplicate consumption, discard
}
// Execute business...
}
}Technical Highlights Analysis
setIfAbsentprovides native atomicity, implementing check-and-set in one step, no concurrency vulnerability.
Key idea: only set key, never delete key; rely on TTL window expiration to auto-clean, perfectly covering duplicate message arrival time window.
Window time must exceed MQ retry cycle (e.g., Kafka's delivery.timeout.ms) to avoid "expired but duplicate still in flight".
3. Optimistic Lock Version Update (State Machine Idempotency)
@Mapper
public interface OrderDao {
// Highlight: versioned update, affected rows auto-deduplicate
@Update("UPDATE orders SET status = #{newStatus}, version = version + 1 " +
"WHERE order_id = #{orderId} AND version = #{oldVersion}")
int updateStatusWithVersion(@Param("orderId") String orderId,
@Param("newStatus") String newStatus,
@Param("oldVersion") int oldVersion);
}
// Business layer call
@Transactional
public void dispatchOrder(String orderId, int currentVersion) {
int rows = orderDao.updateStatusWithVersion(
orderId,
OrderStatus.SHIPPED.getCode(),
currentVersion
);
if (rows == 0) {
// Duplicate message or version conflict, idempotent ignore
throw new DedupSkipException("Order status already changed, skip duplicate operation");
}
// Subsequent shipping logic...
}Technical Highlights Analysis
Uses version field for optimistic locking, simultaneously providing natural idempotency: same message retries with stale version, affected rows = 0.
Compared to WHERE status = ?, prevents concurrent overwrites (two "payment success" messages arriving simultaneously, version causes second to fail).
Suitable for state transition scenarios like "payment → shipping → confirmation", each state change increments version.
Summary of Technical Challenges and Solutions
Unclear Transaction Boundaries – Deduplication mark and business operation not in same transaction, causing stray marks or data inconsistency. Solution: use database transaction or TCC pattern; mark and business must be atomic (see Code 1).
Concurrency Race – Two identical messages consumed simultaneously, both judge "unprocessed", leading to duplicate execution. Solution: ① database unique index constraint; ② Redis SETNX atomicity; ③ version comparison (see Code 2/3).
Mark Cleanup Timing – Premature deletion of idempotency mark allows late duplicate to re-execute. Solution: never actively delete marks ; use expiration time for natural eviction (Redis TTL); database deduplication table can be archived periodically but not deleted.
Unreliable MQ Message ID – MQ resend may generate new messageId, breaking idempotency based on it. Solution: enforce business unique identifier (order number, serial number); never rely on MQ message ID.
Cache Penetration & Idempotency Conflict – Delete cache before business; duplicate finds cache missing and re-runs business. Solution: always set mark before deleting cache , or reverse: cache updated only after business success; mark has highest priority.
Performance Bottleneck Under High Concurrency – Database unique index becomes write hotspot, affecting TPS. Solution: introduce Redis as first-level idempotency (hot data), database as second-level fallback (cold data); sharding and degradation strategies.
Unpredictable Duplicate Arrival – Duplicate may arrive days later, making deduplication window hard to set. Solution: set sufficiently long window based on business tolerance (e.g., payment callback 7 days); database persistent marks support long-tail deduplication.
Rollback and Compensation – Business fails halfway, mark already written, subsequent retry cannot recover. Solution: introduce business state checkpoints ; mark only indicates "message received"; business success tracked by independent state machine or reconciliation system.
Interviewer concludes: "This code and challenge breakdown combines theory and practice well. Especially mark not deleted, business ID deduplication, transaction atomicity are the dividing line for senior engineers. Articulating these details shows you've used them in real projects, not just memorized answers. High score for this scenario question."
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
