8 Practical API Idempotency Solutions to Eliminate Duplicate Requests (Pitfall Guide)

Introduction

"Junior, a user complained that a single payment click resulted in three charges! This bug must be fixed today," the senior developer asks. In modern distributed systems, network jitter, user mis‑operations, and retry mechanisms can cause the same request to be executed multiple times. Idempotency is the key to solving this problem.

What Is Idempotency?

In mathematics, an operation is idempotent when executing it once or many times yields the same result, e.g., f(x) = 1 always returns 1 regardless of how many times it is called. In programming, an API is idempotent when repeated calls with the same parameters produce identical effects and no side‑effects.

Typical example: a payment interface. The ideal case is one click, one deduction, and the order status becomes "Paid". If the interface lacks idempotency, a user may click repeatedly or the payment system may retry, causing multiple deductions while the order may still be "Pending"—a serious financial loss.

Why Do Duplicate Requests Occur?

They arise from several sources:

Client / Frontend Issues

User mis‑operation: rapid double‑click, especially on mobile.

Missing client‑side guard: button not disabled, no loading state.

Script retry: some frameworks automatically retry on timeout.

Network Issues

Network jitter: request reaches the server, response is lost, client retries.

Gateway retry: API gateways or load balancers may be configured to retry.

Backend Microservice Calls

Timeout retry: Service A calls Service B, B is slow, A times out and retries while B has already processed the first request.

Message Queue Replay

When a message is re‑queued after a broker failure (e.g., RabbitMQ requeue), the same logical request can be delivered again.

How to Guarantee Idempotency?

The two core ideas are: (1) identify duplicate requests, and (2) make a duplicate request either a no‑op or return the result of the first execution.

Solution 1: Token Mechanism (Client‑Side Guard)

The client obtains a unique token (usually a UUID) before performing a critical operation, stores it in Redis with a short TTL, and sends the token in the request header Idempotent-Key. The server checks Redis: if the token is missing, the request is rejected; if present, the token is deleted atomically and the business logic proceeds.

Obtain token: call /getToken, store in Redis with 30‑minute expiration.

Send token: include Idempotent-Key: token_value in the request header.

Validate token: check Redis existence, delete atomically, then execute business logic.

@RestController
public class IdempotentController {

    @Autowired
    private RedisTemplate<String, String> redisTemplate;

    // 1. Token acquisition endpoint
    @GetMapping("/getToken")
    public String getToken() {
        String token = UUID.randomUUID().toString();
        redisTemplate.opsForValue().set("idempotent:token:" + token, "1", 30, TimeUnit.MINUTES);
        return token;
    }

    // 2. Idempotent business endpoint
    @PostMapping("/submitOrder")
    public String submitOrder(@RequestHeader("Idempotent-Key") String token, @RequestBody Order order) {
        String redisKey = "idempotent:token:" + token;
        Boolean isExist = redisTemplate.delete(redisKey);
        if (Boolean.TRUE.equals(isExist)) {
            // First request – execute core logic
            return "Order submitted successfully!";
        } else {
            // Duplicate request
            return "Please do not submit the order repeatedly";
        }
    }
}

Advantages : simple implementation, clear logic, low intrusion (can be wrapped with AOP).

Disadvantages : requires an extra request to obtain the token, adds network overhead, token must be stored securely, and token expiration may cause false negatives.

Applicable scenarios : frequent front‑end interactions such as form submission, ordering, payment.

Solution 2: Database Unique Index (Simple DB Guard)

Define a unique constraint on a column (e.g., phone number for registration). When an INSERT violates the constraint, the database throws DuplicateKeyException, which the application catches and treats as a duplicate request.

ALTER TABLE user ADD UNIQUE INDEX uk_phone (phone);

@Service
public class UserService {
    @Autowired
    private UserMapper userMapper;

    public String register(User user) {
        try {
            userMapper.insert(user);
            return "Registration successful";
        } catch (DuplicateKeyException e) {
            log.warn("Phone number already registered: {}", user.getPhone());
            return "This phone number is already registered";
        }
    }
}

Advantages : extremely simple, high performance due to index lookup.

Disadvantages : only protects insert operations; idempotency granularity is limited to the indexed column.

Applicable scenarios : data‑creation APIs such as user registration, order creation with a unique order number.

Solution 3: Optimistic Lock (Version‑Based Guard)

Add a version column to the table. When updating, include the version in the WHERE clause and increment it. If another transaction has already updated the row, the WHERE condition fails, resulting in zero rows affected—indicating a duplicate request.

ALTER TABLE `order` ADD COLUMN `version` INT DEFAULT 0;

UPDATE `order`
SET status = 'paid', version = version + 1
WHERE order_id = '123' AND version = 0;

Application logic checks the affected row count; zero means the request was already processed.

Advantages : avoids database locks, good performance for read‑heavy scenarios.

Disadvantages : high contention leads to many retries; requires an extra column.

Applicable scenarios : update operations with state changes, such as order status updates, inventory deduction (combined with a check that stock > 0).

Solution 4: Pessimistic Lock (Row‑Level Lock)

Use SELECT ... FOR UPDATE inside a transaction to lock the target row. Other transactions wait until the lock is released. SELECT * FROM account WHERE user_id = #{userId} FOR UPDATE; After acquiring the lock, perform the business logic (e.g., balance deduction). The lock is released when the transaction commits.

Advantages : guarantees strong consistency, simple to write.

Disadvantages : lower throughput, cannot by itself prevent duplicate requests—if two requests acquire the lock sequentially, both will execute the business logic.

Applicable scenarios : high‑consistency operations with low concurrency, such as financial transactions where the lock is combined with another idempotency measure.

Solution 5: Distributed Lock (Cross‑JVM Guard)

In a micro‑service environment, use a shared middleware (Redis, ZooKeeper) to create a mutex across JVMs. The service tries to acquire the lock before executing the critical section.

@Service
public class SmsService {
    @Autowired
    private RedissonClient redissonClient;

    public String sendCouponSms(Long userId, String activityId) {
        String lockKey = "sms:lock:" + activityId + ":" + userId;
        RLock lock = redissonClient.getLock(lockKey);
        try {
            boolean isLocked = lock.tryLock(0, 5, TimeUnit.SECONDS);
            if (!isLocked) {
                return "Requests are too frequent, please try later";
            }
            // ... send SMS logic ...
            return "Send successful";
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            return "System error";
        } finally {
            if (lock.isHeldByCurrentThread()) {
                lock.unlock();
            }
        }
    }
}

Advantages : works in distributed environments, high flexibility.

Disadvantages : introduces external component complexity; improper expiration settings can cause deadlocks.

Applicable scenarios : distributed critical sections such as flash‑sale stock deduction, global task scheduling.

Solution 6: State Machine Engine (Complex Workflow)

For business processes with multiple states (e.g., order lifecycle), define states, events, and transition rules. The state machine records the current state; an incoming event is processed only if the transition is allowed. Duplicate events are rejected because the state has already moved.

public enum OrderState { INIT, PAID, DELIVERED, CONFIRMED, CANCELLED }
public enum OrderEvent { PAY_SUCCESS, PAY_FAIL, SHIP, CONFIRM_RECEIPT, CANCEL }

@Service
public class OrderStateService {
    public boolean handleEvent(String orderId, OrderEvent event) {
        OrderState currentState = /* load from DB */;
        boolean success = stateMachine.sendEvent(MessageBuilder.withPayload(event)
                .setHeader("orderId", orderId).build());
        if (success) {
            // persist new state, execute business logic
            return true;
        } else {
            log.warn("Invalid state transition. orderId: {}, currentState: {}, event: {}", orderId, currentState, event);
            return false; // or treat as idempotent success
        }
    }
}

Advantages : clear logic, strong maintainability, built‑in idempotency.

Disadvantages : higher learning curve; may be overkill for simple APIs.

Applicable scenarios : any lifecycle‑driven domain such as orders, work tickets, approval flows.

Solution 7: Request Serial Number (Financial‑Grade Guard)

The caller must provide a globally unique business serial number (e.g., out_trade_no). The server stores the serial number together with the result in a table with a unique index. Subsequent requests with the same serial number return the stored result.

@PostMapping("/pay")
public PayResponse pay(@RequestBody PayRequest request) {
    String outTradeNo = request.getOutTradeNo();
    PayTransaction existing = payTxMapper.selectByOutTradeNo(outTradeNo);
    if (existing != null) {
        return new PayResponse(existing.getStatus(), existing.getTransactionId());
    }
    // core payment logic ...
    try {
        payTxMapper.insert(newPayTx);
    } catch (DuplicateKeyException e) {
        existing = payTxMapper.selectByOutTradeNo(outTradeNo);
        return new PayResponse(existing.getStatus(), existing.getTransactionId());
    }
    return new PayResponse("SUCCESS", newPayTx.getTransactionId());
}

Advantages : extremely reliable, industry standard for payment APIs.

Disadvantages : requires client cooperation and persistent storage of the serial number and result.

Applicable scenarios : all financial transaction APIs and open APIs that need strong idempotency.

Solution 8: MQ Idempotency (Message Replay Guard)

Each message is assigned a globally unique message_id (e.g., UUID). Consumers first check a "message consumption" table. If the message is already marked as consumed, they ACK immediately; otherwise they insert a record, process the business logic, and update the status.

Producer: generate message_id and set it in the message properties.

Consumer: retrieve message_id, query the consumption table.

If status = consumed → ACK.

If status = pending → process, then update to consumed and ACK.

If insert fails (duplicate key) → another consumer already processed → ACK.

Key table fields: id (PK), message_id (UNIQUE), topic, tag, business_id, status (1‑pending, 2‑consumed, 3‑failed), timestamps.

@Component
public class OrderPayMQListener {
    private static final Logger log = LoggerFactory.getLogger(OrderPayMQListener.class);

    @Autowired
    private MqMessageMapper mqMessageMapper;
    @Autowired
    private OrderService orderService;

    @RabbitListener(queues = "order.pay.result")
    public void onMessage(Message message, Channel channel) throws IOException {
        String messageId = message.getMessageProperties().getMessageId();
        if (StringUtils.isEmpty(messageId)) {
            channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, false);
            log.error("MQ message missing messageId, rejected");
            return;
        }
        MqMessageDO existing = mqMessageMapper.selectByMessageId(messageId);
        if (existing != null) {
            if (existing.getStatus() == 2) {
                channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
                log.info("MQ message already consumed, messageId: {}", messageId);
                return;
            } else if (existing.getStatus() == 1) {
                channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, true);
                log.warn("MQ message processing, messageId: {}, will retry", messageId);
                return;
            }
        }
        // Insert consumption record (status = 1)
        MqMessageDO messageDO = new MqMessageDO();
        messageDO.setMessageId(messageId);
        messageDO.setTopic("order.pay.result");
        messageDO.setTag(message.getMessageProperties().getReceivedRoutingKey());
        messageDO.setStatus(1);
        try {
            int rows = mqMessageMapper.insertMessage(messageDO);
            if (rows <= 0) {
                channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, false);
                log.error("Failed to insert MQ record, messageId: {}", messageId);
                return;
            }
        } catch (DuplicateKeyException e) {
            channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
            log.warn("Concurrent MQ consumption, messageId: {}", messageId);
            return;
        }
        // Business processing
        String body = new String(message.getBody(), StandardCharsets.UTF_8);
        PayResultDTO payResult = JSON.parseObject(body, PayResultDTO.class);
        Result result = orderService.syncPayResult(payResult);
        if (result.isSuccess()) {
            mqMessageMapper.updateMessageConsumed(messageDO.getId(), payResult.getOrderId());
            channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
            log.info("MQ consumed successfully, messageId: {}, orderId: {}", messageId, payResult.getOrderId());
        } else {
            mqMessageMapper.updateMessageFailed(messageDO.getId());
            channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, false);
            log.error("MQ consumption failed, messageId: {}, reason: {}", messageId, result.getMsg());
        }
    }
}

Advantages : fits MQ scenarios, supports retry mechanisms, provides traceability.

Disadvantages : depends on a database table, adds latency.

Applicable scenarios : any MQ‑driven business logic such as payment callbacks, asynchronous notifications.

Pitfall Guide

Pitfall 1 : Generating messageId from business content (e.g., order number) can cause collisions. Use a globally unique UUID.

Pitfall 2 : Using automatic ACK causes messages to be considered consumed before business success. Always use manual ACK.

Pitfall 3 : Unlimited retry loops waste resources. Set a maximum retry count and route overflow to a dead‑letter queue.

Decision Matrix Summary

1. Token Mechanism : client‑side one‑time token; suitable for front‑end actions; simple AOP integration; adds extra request.

2. Unique Index : database constraint; ideal for insert‑only operations; highly reliable; limited to uniqueness of indexed column.

3. Optimistic Lock : version field; good for high‑read, low‑write scenarios; may cause many retries under contention.

4. Pessimistic Lock : row‑level lock; guarantees strong consistency; lower throughput; does not alone prevent retries.

5. Distributed Lock : cross‑JVM mutex (Redis/ZooKeeper); fits distributed critical sections; adds complexity and risk of deadlocks.

6. State Machine : explicit state‑event rules; perfect for complex workflows; higher learning cost.

7. Request Serial Number : globally unique business ID; industry‑standard for payments; requires client cooperation.

8. MQ Idempotency : message‑ID + consumption record; tailored for MQ replay; introduces DB latency.

Best‑Practice Recommendations

Choose an idempotency key that uniquely identifies a business operation (e.g., token, user‑ID + action, order number, serial number).

Apply multi‑layer defense: front‑end button disable, gateway/token validation, database or lock protection, and, if needed, a state‑machine guard.

Remember that "at‑least‑once" delivery plus idempotent processing yields "exactly‑once" semantics.

Design APIs with idempotency in mind: use PUT for create‑or‑replace, require unique IDs for POST, and employ optimistic locking for updates.

Return a clear response for duplicate requests—usually the same success payload as the first request, unless the request is malicious, in which case a 4xx error may be appropriate.