Mastering Idempotency: Design Patterns and Code Solutions for Reliable APIs

Article Overview

Ancient saying: "Once is enough, twice is the same" – the effect remains unchanged no matter how many times an operation is performed. This is idempotency.

What Is Idempotency?

Interface idempotency means that a user’s same operation results in consistent outcomes whether the request is sent once or multiple times, without side effects. For example, a bus card swipe should not deduct money twice if the user clicks the button repeatedly.

Note: Idempotency issues can arise in databases, but they are not limited to databases.

Scenarios Requiring Idempotent Design

High‑value or consistency‑critical scenarios: Online Payment: Prevent duplicate charges. Bank Transaction: Ensure a transaction is not executed multiple times. Ticketing System: Avoid double‑booking seats. Communication Service: Prevent duplicate billing for SMS or calls. Task Scheduling: Avoid re‑executing the same job on retries. User Registration: Prevent creating duplicate user records.

How Idempotency Problems Occur

Common causes: Network request retries due to timeouts or instability. UI duplicate submissions from accidental double‑clicks. Message queue retry mechanisms (e.g., Kafka, RabbitMQ) causing duplicate consumption. Database concurrent operations without proper locking or isolation. External API retries that repeat database actions.

Other edge cases.

Case Study: Order Table Design

Design a sample orders table:

Table Structure

Field Descriptions

order_id : Global unique identifier (UUID, Snowflake, etc.).

user_id : Links to the user.

product_id : Links to the product.

quantity : Number of items.

order_status : Controls business flow; e.g., payment allowed only when status is "pending".

create_time : Timestamp of order creation.

pay_time : Timestamp of payment.

version : Optimistic‑lock version number.

Business Rules

Order Payment : Check order_status is "pending" before paying, then set to "paid".

Order Cancellation : Allow cancellation only in specific states.

Insert Order : Use order_id as a unique constraint to prevent duplicate inserts.

Optimistic Lock : Compare stored version with the current version; reject update if they differ.

Idempotency Solutions

Common techniques in distributed systems:

1. Unique Constraints

Leverage database unique indexes or primary keys to block duplicate inserts.

INSERT INTO `mydb`.`orders` (`order_id`,`user_id`,`product_id`,`quantity`,`order_status`,`create_time`,`pay_time`,`version`)
VALUES ('ORD-20231023-0001','USR-A123456','PRD-X123',2,0,'2023-10-23 10:15:30',NULL,1);
-- ERROR 1062 (23000): Duplicate entry 'ORD-20231023-0001' for key 'orders.PRIMARY'

2. Optimistic Lock

Store a version number or timestamp; update only when the stored version matches.

UPDATE orders
SET quantity = 1,
    order_status = 1,
    pay_time = '2024-04-30 10:20:00',
    version = version + 1
WHERE order_id = 'ORD-20231023-0001' AND version = 1;

3. Pessimistic Lock

Lock rows during a transaction using SELECT ... FOR UPDATE.

-- Lock the record
SELECT * FROM orders WHERE order_id = 'ORD-20231023-0001' FOR UPDATE;

-- Execute business logic
UPDATE orders SET quantity = 1, order_status = 1, pay_time = '2023-10-23 10:20:00' WHERE order_id = 'ORD-20231025-0003';

4. Distributed Lock

Use a Redis‑based lock to ensure only one instance processes a request.

public class MyService {
    private final RedisDistributedLock lock;
    public MyService(Jedis jedis, String lockKey, int lockTimeout) {
        this.lock = new RedisDistributedLock(jedis, lockKey, lockTimeout);
    }
    public void executeInLock() {
        if (lock.tryLock()) {
            try {
                // business logic
            } finally {
                lock.unlock();
            }
        } else {
            // handle lock acquisition failure
        }
    }
}

It is recommended to use a Lua script for atomic lock release.

String unlockScript = "if redis.call('get', KEYS[1]) == ARGV[1] then " +
                       "return redis.call('del', KEYS[1]) " +
                       "else return 0 end";
jedis.eval(unlockScript, 1, lockKey, "1");

5. Token Mechanism

Generate a unique token per request, store it in Redis, and delete it after processing to prevent repeats.

void do(String token) {
    if (Redis.exists(token)) {
        Redis.del(token);
        // execute business logic
    } else {
        // duplicate request handling
    }
}

6. State Machine

Model business flow with states (e.g., PENDING → PAID) to reject repeated actions.

public enum OrderStatus { PENDING, PAID, CANCELLED }
public class Order {
    private OrderStatus status = OrderStatus.PENDING;
    public synchronized void pay() {
        if (status == OrderStatus.PENDING) {
            // payment logic
            status = OrderStatus.PAID;
        } else {
            throw new IllegalStateException("Order can only be paid when status is PENDING");
        }
    }
}

7. Deduplication Table

Record processed request IDs; skip processing if the ID already exists. Apply expiration policies to clean old records.

8. Global Request ID

Assign a unique ID to each request (e.g., via Nginx) and store it in Redis sets for deduplication.

proxy_set_header X-Request-Id $request_id;

Conclusion

By applying the above strategies—unique constraints, optimistic/pessimistic locks, distributed locks, token mechanisms, state machines, deduplication tables, and global request IDs—developers can achieve reliable idempotent behavior in high‑concurrency systems.