Async Idempotency in Practice: Local Message Tables, Duplicate Deduction Fixes, and Layered Defense

01. Incident Recap: Duplicate Coupon Deduction During Peak Promotion

During last year's major promotion, the local-message-table-based eventual consistency pipeline passed unit, integration, and single-link load tests. After 40 minutes at peak traffic, coupon reconciliation alerts fired: some users had two coupons deducted for a single order.

Initial investigation of the message table state machine (three states: pending, success, failed) showed no logic gaps. Database binlog analysis revealed the root cause: the message was ultimately marked success, but the coupon deduction logic executed twice.

Root cause: The business deduction succeeded and the transaction committed, but a millisecond-level network blip caused the DB's commit ACK packet to be lost. The application timed out, leaving the message in pending state. The scheduled polling task then retried the consumption logic, causing a second deduction. This is the classic "uncertain result" problem in distributed systems: the DB transaction persisted, but the application could not confirm the outcome.

Key insight: The retry self-healing capability of the local message table is itself the source of duplicate execution. Retry is not an add-on; it is the foundation on which this consistency scheme survives. Once retry is enabled, duplicate execution must be accepted as inevitable.

Many teams focus solely on message state transitions and transaction choreography, assuming a correct state machine guarantees eventual consistency. In async links, duplicate traffic originates from MQ master failover, payment platform callback retries, and local message polling — internal, system-generated duplicates that frontend debouncing or one-time tokens cannot intercept. This is the fundamental boundary between async and synchronous idempotency.

02. Mechanism Analysis: Sources and Characteristics of Duplicate Traffic in Async Scenarios

Synchronous duplicate requests come from end users and can be blocked at the gateway. Async duplicate traffic is an inherent byproduct of unreliable networks in distributed systems; it cannot be eliminated, only its side effects isolated.

1. Local message table retry: Business transaction commits, commit receipt lost, app catches timeout, message stays pending. Scheduled task scans and retries the entire business logic.

2. RocketMQ message replay: Broker master crashes, cluster fails over, consumer hasn't committed offset. New master redelivers already-consumed but unacknowledged messages.

3. Third-party payment callbacks: Payment platform's retry policy is a safety net; any non-200 response or connection timeout triggers continuous callback pushes regardless of whether order-side deduction/shipment completed.

First major pitfall: Many teams try to eliminate duplicate traffic at the source. Disabling retry avoids duplicates but directly introduces message loss and data inconsistency. Retry is fault self-healing; idempotency is its mandatory constraint. They coexist and cannot be traded off independently. Idempotency's true goal is not to intercept requests, but to decide — based on business domain state — whether the action has already been persisted; if so, skip the business change and return success.

03. Engineering Pitfalls: Trade-offs and Costs of Four Idempotency Solutions

Solution 1: Database Unique Index

Engineers often prefer a unique constraint on a business unique identifier; duplicate inserts throw a unique conflict exception.

Pros: Native DB guarantee, minimal implementation cost, little business code change.

Hidden engineering costs: Under high-concurrency retry, multiple concurrent requests contend for the same unique index row, causing massive row-lock waits that amplify DB pressure during peak promotions. In sharded architectures, the idempotent record must reside in the same shard as the business data; cross-shard scenarios cannot reuse this approach. Critical blind spot: Unique indexes only guard insert-type writes; they cannot protect state-change operations . Repeat execution of order or coupon status updates will not trigger a unique conflict and will still repeat the business action.

Verdict: Limited use for creating flow records or documents; absolutely cannot be the sole reliance for inventory deduction, coupons, or state transitions.

Solution 2: Idempotent Record Table (Business Flow Idempotency)

Standalone idempotent table storing global idempotent key, business type, business document number. Before executing business logic, query the idempotent record.

Standard flow: Start transaction → Query idempotent record → If absent, execute business logic and write idempotent record → Commit transaction.

Fatal race condition: Query and write are non-atomic. Without locking, two retry requests query simultaneously, both find no record, both execute business logic, causing duplicate deduction. Fix: Place query, business execution, and idempotent record write in the same transaction, protected by a unique index on the idempotent table to eliminate concurrent races.

Long-term maintenance debt: The idempotent table grows continuously; during promotions data explodes. Without archiving/cleanup, half a year can yield 100M+ rows, degrading lookup latency and slowing the entire async pipeline.

Implementation standard: Add expire_time column; offline-archive data beyond the retention window to a history DB; keep only the most recent N days in the online business DB.

CREATE TABLE biz_idempotent_record (
  id BIGINT AUTO_INCREMENT PRIMARY KEY COMMENT 'primary key',
  idempotent_key VARCHAR(128) NOT NULL COMMENT 'global idempotent key: msgId/payment serial',
  biz_type TINYINT NOT NULL COMMENT 'business type: coupon/order/inventory',
  biz_no VARCHAR(64) NOT NULL COMMENT 'business document number',
  create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
  expire_time DATETIME NOT NULL COMMENT 'expiry/archive time',
  UNIQUE KEY uk_idempotent_key (idempotent_key)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Note: The unique index solves the race condition; expire_time — omitted by most designs — enables offline archiving to prevent unbounded table growth.

Solution 3: Business State Machine Idempotency (Mall's Primary Approach)

Domain entities with full lifecycles (orders, coupons, inventory) can leverage their own domain state for idempotency. Before executing an action, verify the domain object's current state; only if it is in an allowed transition state, perform the change.

Coupon deduction rule: Only [UNUSED] coupons may be deducted; if state is already [DEDUCTED] , return success immediately, skipping deduction logic.

❌ Production Fault Version (Race Condition)

// Fault version: state check outside transaction, concurrent retry causes race
public void consumeCouponMsg(String msgId, Long couponId){
  // Query OUTSIDE transaction! Concurrent retries read stale snapshot
  Coupon coupon = couponMapper.selectById(couponId);
  if(!CouponStatus.UNUSED.equals(coupon.getStatus())){
    log.info("Duplicate message, coupon already deducted msgId:{}", msgId);
    return;
  }
  // ERROR: business op and state check separated
  couponMapper.deductCoupon(couponId, CouponStatus.USED);
  localMsgMapper.updateStatus(msgId, MsgStatus.SUCCESS);
}

Postmortem: The core issue is not annotation style but state verification detached from transaction isolation . Unit tests and normal traffic hide the bug; only peak concurrent retry triggers it. Two retry threads query almost simultaneously, both read UNUSED, both enter transaction. DB row lock serializes two deduction statements, resulting in double deduction. Red line: Domain state verification and business mutation must be wrapped in the same DB transaction . Detached verification guarantees a race condition under concurrency.

✅ Fixed Core Business Code

@Transactional(rollbackFor = Exception.class)
public void consumeCouponMsg(String msgId, Long couponId){
  // 1. Business state verification (all inside transaction)
  Coupon coupon = couponMapper.selectById(couponId);
  if(!CouponStatus.UNUSED.equals(coupon.getStatus())){
    log.info("Duplicate message, coupon deduction already completed msgId:{} couponId:{}", msgId, couponId);
    return;
  }
  // 2. Execute business deduction + update business state (atomic in transaction)
  int rows = couponMapper.deductCoupon(couponId, CouponStatus.USED);
  if(rows <= 0){
    throw new BizException("Coupon deduction failed, no rows affected for state change");
  }
  // 3. Update local message table status, mark completed
  localMsgMapper.updateStatus(msgId, MsgStatus.SUCCESS);
}

How it works: State check, deduction, and message status update are in one local transaction. If commit-phase network blip occurs and app receives timeout, the DB transaction actually succeeded. Scheduled retry re-enters the method, sees coupon state is USED, returns immediately — no second deduction. Advantages: No extra idempotent table, no extra storage overhead, semantics align with domain model, debugging is intuitive. Boundary constraints: Only applies to domain entities with explicit lifecycles and state transitions; stateless operations like pure notifications/pushes cannot use it. State verification and mutation must not be split; splitting reintroduces the race condition.

Trade-off cost: State machine idempotency is not universal. For pure notification scenarios (SMS, in-app messages) where no business entity state exists, forcing a state machine pollutes the domain model; an independent idempotent record table fits better. State machine idempotency also demands a complete state lifecycle in the domain model, raising domain modeling complexity.

Solution 4: Token Anti-Replay (Synchronous-Only, Useless in Async Links)

One-time token schemes target frontend APIs: user fetches token, backend consumes it once for deduplication.

But MQ replay, third-party payment callbacks, and other backend async traffic never carry a frontend one-time token. Many teams mistakenly reuse synchronous anti-replay in async links; idempotency protection completely fails. Token schemes have zero protective power in async links and must not be mixed in.

04. Hidden Control-Plane Risks: Overlooked Idempotency System Defects

Risk 1: Idempotent Key Design Flaws

Idempotent key must be globally unique, semantically stable, and invariant across retries .

Wrong choices: auto-increment IDs, service-local random strings; retries generate new values, losing deduplication.

Correct choices: business document number, payment platform callback serial, MQ message msgId + topic combination.

Extra caution: Externally supplied IDs must be validated for legitimacy to prevent malicious crafted IDs bypassing idempotency logic.

Risk 2: Idempotency Logic Timeout and Compensation Boundaries

Passing idempotent check ≠ business definitely landed. If the process crashes mid-execution after idempotent record write but before business mutation, subsequent retries see the idempotent mark and skip, creating a permanent data gap.

This is the classic distributed dilemma: write idempotent record first → mark exists but business not done; write idempotent record last → concurrent retries cannot be blocked.

Solution: Idempotent record, business data, and message status commit in the same local transaction . The local message table pattern satisfies this condition, which is why pairing it with business state machine idempotency became the mall's primary combination.

Risk 3: Cache-Based Idempotency Penetration Risk

Some teams store idempotent markers in Redis for fast lookup.

Danger: Redis cluster outage, key expiry, network blip → cache query exception → business proceeds → duplicate execution. Redis cache is only suitable for front-end fast interception ; it must never be the sole idempotent data source . The bottom line must be DB-persisted fallback. Cache is a performance optimization, not a replacement for durable storage.

05. Multi-Layer Architecture Closure: Layered Idempotency Design for Async Links

Don't try to cover all traffic entry points with one idempotency scheme. The mall's production architecture uses a layered defense strategy, matching each entry point with its appropriate deduplication mechanism to form a multi-layer protection closure:

1. MQ consumption link: msgId + topic as idempotent key, combined with business state machine verification; Redis front-end fast check, DB as ultimate fallback.

2. Third-party payment callbacks: Payment platform transaction serial as idempotent key, plus order business state secondary verification.

3. Local message table retry link: Relies on business domain state machine, coupled with message table status verification — the core protection layer for this scenario.

Core philosophy of layered design: Outer layer fast interception, inner layer durable fallback. If outer cache fails, DB layer holds the last line; the business never gets penetrated directly. Layering is not reinventing the wheel — it's fault-domain isolation, preventing a single component failure from collapsing the entire anti-duplicate system.

06. Implementation Review Checklist (Ready for Architecture Reviews)

Is the idempotent key globally unique? Are externally supplied IDs validated for legitimacy?

Are idempotent verification and business mutation wrapped in the same DB transaction?

Have concurrent race scenarios been assessed? Any non-atomic query-then-write vulnerabilities?

Does the idempotent record table have an archiving/expiry cleanup strategy? Long-term storage cost evaluated?

Are traffic entry points distinguished (MQ / callback / local message retry) with matched idempotency schemes?

Is cache idempotency only a front-end optimization? Any case where cache is the sole data source?

Failure contingency: Are data repair scripts prepared for idempotency logic anomalies or dirty data?

Load test verification: Dedicated concurrency retry stress tests to validate idempotency effectiveness under high concurrency.

07. Architectural Elevation Summary

Many architects treat idempotency as a mere dev-level utility — add a unique index, write a few conditional checks, call it done. From a distributed async system perspective, idempotency is the mandatory constraint layer paired with the distributed retry self-healing mechanism .

The local message table relies on retry to solve message loss and data inconsistency; retry inherently introduces duplicate execution risk. Idempotency is not an independent optional feature; it is an inseparable link in the entire eventual consistency system.

No universal idempotency solution exists. Unique index, independent idempotent table, business state machine — each has its applicability boundary, performance overhead, and long-term maintenance debt.

Core of architectural trade-off: Not pursuing the strongest anti-duplicate capability, but drawing clear boundaries by combining business domain model, traffic peak characteristics, and storage costs to build a layered, controllable protection closure. Blindly stacking idempotency logic without considering business context only increases system complexity and buries hidden technical debt of data deadlocks and hard-to-repair failures. Distributed architecture design is essentially finding controllable balance points among inherent contradictions.