How to Build a Robust Idempotent Framework for Distributed Systems

Background

Answering a group member's question: Is there a universal solution and practice for idempotency?

The article assumes readers are familiar with the concept of idempotency and have encountered related problems in their own projects.

While core business logic (e.g., order payment) often handles idempotency internally, many non‑core scenarios such as network retries or repeated user clicks also require a generic idempotent framework to simplify development.

Simple Idempotent Implementation

Database Record Judgment

Using the payment scenario as an example, an extra table records each successful action. Before processing a payment, the system checks whether a record for the order already exists; if it does, the request is treated as already successful, achieving idempotency.

Concurrency Problem Solution

When two requests for the same order are processed concurrently, both may read an empty record and proceed, causing duplicate payments. Simple solutions include a unique index on the record table or a distributed lock that ensures only one request can modify the resource at a time.

Generic Idempotent Implementation

To let developers focus on business logic, a reusable idempotent component is provided. The following sections describe its design.

Design Scheme

Generic Storage

The framework first queries a storage layer and then performs the appropriate action. A lock is always applied to avoid concurrent conflicts, while the storage records whether the operation has already been executed.

Ease of Use

Developers inject the idempotent component and call it directly, without dealing with locking or record‑checking logic.

Annotation Support

An annotation can be placed on business methods; the framework automatically handles locking, record checking, and duplicate‑request handling.

Multi‑level Storage

Level‑1 storage (e.g., Redis) provides high‑performance short‑term caching, while Level‑2 storage (e.g., MySQL, MongoDB) offers durable long‑term persistence. The storage type is configurable, and a strategy pattern is used to select the appropriate implementation.

Concurrent Read/Write

Two modes are supported: sequential writes (first to Level‑1, then Level‑2) and parallel writes (simultaneous writes to both levels) to improve performance.

Idempotent Execution Flow

Idempotent Interface

public interface DistributedIdempotent {
    /**
     * Idempotent execution
     * @param key Idempotent key
     * @param lockExpireTime Lock expiration time
     * @param firstLevelExpireTime First‑level storage expiration
     * @param secondLevelExpireTime Second‑level storage expiration
     * @param timeUnit Time unit for expiration
     * @param readWriteType Read/write mode
     * @param execute Business logic to execute
     * @param fail Logic to run when the key already exists
     * @return Result of execution
     */
    <T> T execute(String key, int lockExpireTime, int firstLevelExpireTime, int secondLevelExpireTime, TimeUnit timeUnit, ReadWriteTypeEnum readWriteType, Supplier<T> execute, Supplier<T> fail);
}

Usage Example (Code‑Based Idempotency)

/**
 * Idempotent method with return value
 */
public String idempotentCode(String key) {
    return distributedIdempotent.execute(key, 10, 10, 50, TimeUnit.SECONDS, ReadWriteTypeEnum.ORDER, () -> {
        System.out.println("Processing...");
        return "success";
    }, () -> {
        System.out.println("Duplicate request");
        return "fail";
    });
}

Idempotent Annotation

@Idempotent(spelKey = "#key", idempotentHandler = "idempotentHandler", readWriteType = ReadWriteTypeEnum.PARALLEL, secondLevelExpireTime = 60)
public void idempotent(String key) {
    System.out.println("Processing...");
}

public void idempotentHandler(String key, IdempotentException e) {
    System.out.println(key + ": idempotentHandler already executed");
}

Automatic Duplicate Request Differentiation

Both code‑based and annotation‑based approaches require a unique idempotent key. The framework can automatically generate this key by hashing the request URL, parameters, body, and selected headers (e.g., userId). If no explicit key is provided, the generated key is used.

Storage Structure

Redis: store the idempotent key as a simple string with a default value of 1.

MySQL: a dedicated table records keys and expiration times.

CREATE TABLE `idempotent_record` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'Primary key',
  `key` varchar(50) NULL DEFAULT '',
  `value` varchar(50) NOT NULL DEFAULT '',
  `expireTime` timestamp NOT NULL COMMENT 'Expiration time',
  `addTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation time',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Idempotent records';

MongoDB: stores the same fields as JSON documents; collections are created automatically.

Source Code

Source repository: https://github.com/yinjihuan/kitty

Sample projects: https://github.com/yinjihuan/kitty-samples