Ensuring Idempotency in Distributed Systems: Patterns, Code, and Best Practices

Article Overview

What Is Idempotency?

Idempotency means that performing the same operation once or multiple times yields the same result without side effects. In API design, a request that is sent repeatedly should produce identical outcomes, preventing issues such as double charging when a user clicks a payment button multiple times.

Note: Idempotency problems can arise in databases, but they are not limited to database operations.

Scenarios Requiring Idempotent Design

High‑risk scenarios where data consistency is critical include: Online Payment: Prevent duplicate charges when a user initiates payment. Bank Transaction: Ensure a transaction is not executed multiple times due to retries. Ticketing System: Avoid double‑booking of seats. Communication Service: Prevent duplicate billing for SMS or call services. Task Scheduling: Ensure scheduled or batch jobs are not executed repeatedly after a restart. User Registration: Prevent creation of duplicate user records from repeated form submissions.

How Idempotency Issues Arise

Common causes include: Network Request Retry: Clients resend the same request after a timeout. UI Duplicate Submission: Users click a button multiple times. Message Queue Retry: Messages may be consumed more than once. Database Concurrency: Concurrent inserts/updates without proper locking. External API Retry: Third‑party services retry calls, causing duplicate operations.

Other factors.

Table Design (Preparation)

Design an orders table to illustrate idempotent handling.

Table Structure

Field Descriptions

order_id : Global unique identifier (UUID or Snowflake).

user_id : Links to the user who placed the order.

product_id : Links to the purchased product.

quantity : Number of items purchased.

order_status : Current status; only "Pending" orders can be paid.

create_time : Timestamp of order creation.

pay_time : Timestamp of payment.

version : Optimistic‑lock version number.

Business Rules

Order Payment : Check order_status is "Pending" before paying, then set to "Paid".

Order Cancellation : Allow cancellation only in specific states.

Insert Order : Use order_id as a unique constraint to prevent duplicate inserts.

Optimistic Lock : Update only if the stored version matches the current version, then increment it.

Data State Diagram

Idempotency Solutions

Common techniques for achieving idempotency in distributed systems are:

1. Unique Constraints

Leverage database unique indexes or primary keys to reject duplicate inserts.

INSERT INTO `mydb`.`orders` (`order_id`,`user_id`,`product_id`,`quantity`,`order_status`,`create_time`,`pay_time`,`version`)
VALUES ('ORD-20231023-0001','USR-A123456','PRD-X123',2,0,'2023-10-23 10:15:30',NULL,1);
-- ERROR 1062 (23000): Duplicate entry 'ORD-20231023-0001' for key 'orders.PRIMARY'

2. Optimistic Lock

Update rows only when the version matches, then increment the version.

UPDATE orders
SET quantity = 1,
    order_status = 1,
    pay_time = '2024-04-30 10:20:00',
    version = version + 1
WHERE order_id = 'ORD-20231023-0001' AND version = 1;

3. Pessimistic Lock

Lock rows during a transaction using SELECT ... FOR UPDATE to prevent concurrent modifications.

-- Lock the record
SELECT * FROM orders WHERE order_id = 'ORD-20231023-0001' FOR UPDATE;

-- Execute business logic
UPDATE orders SET quantity = 1, order_status = 1, pay_time = '2023-10-23 10:20:00' WHERE order_id = 'ORD-20231025-0003';

4. Distributed Lock

Use a distributed lock (e.g., Redis) so that only one instance processes a specific request.

public class MyService {
    private final RedisDistributedLock lock;
    public MyService(Jedis jedis, String lockKey, int lockTimeout) {
        this.lock = new RedisDistributedLock(jedis, lockKey, lockTimeout);
    }
    public void executeInLock() {
        if (lock.tryLock()) {
            try {
                // business logic
            } finally {
                lock.unlock();
            }
        } else {
            // handle lock acquisition failure
        }
    }
}

It is recommended to use a Lua script to delete the lock atomically.

String unlockScript = "if redis.call('get', KEYS[1]) == ARGV[1] then " +
                       "return redis.call('del', KEYS[1]) " +
                       "else " +
                       "return 0 " +
                       "end";
jedis.eval(unlockScript, 1, lockKey, "1");

5. Token Mechanism

Generate a unique token per request, store it in Redis, and delete it after processing to guarantee single execution.

void do(String token) {
    if (Redis.exists(token)) {
        Redis.del(token);
        // execute business logic
    } else {
        // duplicate request handling
    }
}

Use a Lua script for atomic check‑and‑delete.

if redis.call('get', KEYS[1]) == ARGV[1] then
    return redis.call('del', KEYS[1])
else
    return 0
end

6. State Machine

Model business processes as states; only allow transitions that preserve idempotency.

public enum OrderStatus { PENDING, PAID, CANCELLED }
public class Order {
    private OrderStatus status = OrderStatus.PENDING;
    public synchronized void pay() {
        if (this.status == OrderStatus.PENDING) {
            // payment logic
            this.status = OrderStatus.PAID;
        } else {
            throw new IllegalStateException("Order can only be paid when status is PENDING");
        }
    }
}

7. Deduplication Table

Maintain a table of processed request identifiers and clean it up periodically.

boolean isDuplicate = checkDuplicateInDatabase(requestId);
if (isDuplicate) {
    return previousResult;
} else {
    // business logic
    saveRecord(requestId);
    return newResult;
}

8. Global Request Unique ID

Assign a globally unique ID to each request (e.g., via Nginx) and store it in Redis to filter duplicates.

proxy_set_header X-Request-Id $request_id;