Idempotency Design Principles and Practices in Distributed Systems

Preface

Idempotency is a crucial guarantee for data consistency and safety in distributed systems, particularly in finance and payment domains, where it serves as a hard metric for loss prevention and is reflected in system architecture design.

Definition of Idempotency

Idempotency originates from mathematics, where a function satisfies f(x) = f(f(x)) (e.g., absolute value). In computer science, it means that invoking the same request once or multiple times yields the same result on the server, and the request is processed at most once.

Importance of Idempotency

Consider a payment company allocating funds through a clearinghouse. If the clearinghouse’s response is lost due to a network timeout, the payment company must decide whether to retry without causing duplicate fund transfers. Proper idempotency control enables safe retries and prevents duplicate allocations.

Remote calls can end in three states: success, failure, or unknown (typically caused by timeout or packet loss). In the unknown case, a compensation query can determine whether to retry, or the clearinghouse can enforce idempotency so the caller can retry blindly.

Idempotency is widely applied in architecture design, such as preventing duplicate consumption in message queues and avoiding duplicate form submissions.

Idempotency Design

Two Core Elements

Idempotency consists of an idempotency key and the critical request parameters.

Idempotency key: A unique constraint on the server, usually composed of an upstream idempotent number and its source. API documentation should specify the composition of the key, which uniquely identifies the request.

Critical request parameters: Core business data such as payer account, payee account, amount, currency, and product quantity. These must remain unchanged for the same idempotency key; any change requires a new key.

Idempotency Principles

Caller Must Ensure Uniqueness and Immutability of the Idempotency Key

Explanation

The caller must guarantee that the idempotency key is not duplicated and remains constant for the same business document, regardless of how many times the request is sent.

Anti‑example

Idempotency key duplication can occur due to sequence cycle issues, mismatched step sizes, or cross‑region/partition collisions.

Sequence cycle problem: insufficient evaluation of business volume versus sequence growth leads to duplicate keys.

Sequence step size or segment configuration issues cause cross‑region/partition key collisions.

Idempotency key changes can happen when a transaction generates a key, times out, rolls back locally, and a subsequent request generates a new key.

Generating the key within a transaction, then retrying after a timeout creates a new key, causing potential loss.

Caller Must Keep Critical Business Request Parameters Immutable

Explanation

If the server does not return a result, the caller must not modify the critical business parameters.

Anti‑example

An initial request times out, but the server has already processed it. The client changes the amount and retries, leading to mismatched processing.

Caller Must Not Construct Idempotency Keys Purely In‑Memory Without Persistence

Explanation

Non‑persistent idempotency keys make audit and reconciliation difficult; persistence is a basic requirement.

Anti‑example

// In‑memory concatenated idempotency key
request.setRequestId(BizTypeEnum.getPrefix(xxxDO.getBizType()) + xxxDO.getId());

Idempotency Key Generation Must Not Include RPC Calls Within the Same Transaction

Anti‑example

transactionTemplate.execute(status -> {
    // generate serial number xxx
    SerialDO serialDO = buildSerialDO();
    serialDAO.insert(serialDO);
    someDAO.update(someDO);
    // dubbo RPC call, using xxxId as idempotency key
    invokeRpc(request);
    return true;
});

Correct Example

Place RPC calls outside the transaction.

transactionTemplate.execute(status -> {
    // generate serial number xxx
    SerialDO serialDO = buildSerialDO();
    serialDAO.insert(serialDO);
    someDAO.update(someDO);
    return true;
});
// RPC call after transaction commit
invokeRpc(request);

Use a transaction synchronizer to register a callback that runs after commit; if the transaction is absent, run immediately.

/*** Outer transaction already started ***/
public static void execute() {
    // update document status
    Runnable runnable = () -> {
        response = dubboService.call(request);
    };
    register(runnable);
}
public static void register(Runnable runnable) {
    if (TransactionSynchronizationManager.isActualTransactionActive()) {
        TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronizationAdapter() {
            @Override
            public void afterCommit() {
                runnable.run();
            }
        });
    } else {
        LOGGER.debug("No active transaction.");
        runnable.run();
    }
}

Server Should Not Rely Solely on Queries for Idempotency

Explanation

In distributed concurrent scenarios, a pure query cannot guarantee idempotent inserts. Common guarantees include:

Database constraint: unique index on the idempotency key.

Distributed lock (e.g., Redis, Zookeeper). Not recommended for strong consistency financial scenarios.

Anti‑example

RPC timeout causes local transaction rollback; a retry generates a new idempotency key, leading to loss.

Server Must Ensure Consistent Acceptance Results

Explanation

For the same request, the server should accept it only once and return the same result each time.

Anti‑example

In a refund scenario, the first request is accepted, but the client times out and retries; the server then rejects due to insufficient funds, causing inconsistency.

// Basic validation
// Pessimistic lock to check refundable amount
Assert.isTrue(refundable(xxx), "cannot refund");
// Business logic
try {
    process(xxx);
} catch (Exception e) {
    // idempotency handling
}

Caller Should Compare Critical Business Parameters After Receiving Server Idempotent Result

Explanation

Clients should verify that key parameters (e.g., account, amount) in the server’s response match the original request.

Anti‑example

If the server only checks the idempotency key and ignores parameter changes, a tampered amount could cause loss.

Correct Example

Server returns the already processed business information based on the idempotency key.

Client validates the returned data against expectations.

Summary

The above rules are distilled from historical projects and internet experience, focusing on idempotency design principles. Various implementation techniques such as idempotent tables, optimistic locks, and pessimistic locks exist, but are not detailed here.