How to Ensure Idempotency in Distributed Systems: Practical Solutions

Concept

Duplicate requests occur when the same request is submitted multiple times due to reasons such as rapid user clicks, Nginx retry mechanisms, service‑framework retries, MQ message re‑consumption, or repeated asynchronous callbacks from third‑party payments.

Idempotency means that identical request parameters produce the same result across multiple executions, covering both concurrent duplicate requests and business retries.

Basic Principle

Idempotency requires two conditions: (1) identical request parameters (whether concurrent or not) and (2) consistent results for each execution. In practice, this often involves concurrency control and ensuring the returned outcome is the same, which may be a success response or a failure depending on the scenario.

Causes

1) Network uncertainty in distributed systems (success, failure, unknown) leads third‑party systems to retry. 2) User repeated submissions or system retry mechanisms cause multiple requests.

Common Solutions

The general approach is concurrency control combined with returning the same result.

Unique Index : Use a unique constraint (e.g., order number) so the same value can be inserted only once. Suitable for single‑table insert scenarios.

Select + Insert/Update : Query first, then decide whether to insert or update based on the result. Works for low‑concurrency insert or update cases.

Optimistic Lock : Include a version or status field in the WHERE clause; if the condition fails, the update is rejected. Handles ABA problems and fits non‑high‑concurrency updates with version control.

Anti‑Repeat Token : Generate a token on page load, submit it with the request, and let the server verify its existence. After processing, the token can be marked used or deleted.

Anti‑Repeat Table : Create a dedicated table with a unique field to guarantee no duplicate processing. Useful for simple distributed scenarios where adding extra fields to existing tables is undesirable.

Distributed Lock : Acquire a lock using a unique key before processing. If the lock already exists, the request is considered duplicate and ignored. Ideal for high‑concurrency distributed environments.

Application Example

Consider an order workflow with four stages: order submission (add), payment success, goods dispatch, and receipt confirmation (all updates). In a single‑machine setup, unique indexes or Select+Insert can handle order submission, while optimistic locks manage updates. In distributed environments, Redis‑based distributed locks, anti‑repeat tokens, or anti‑repeat tables are employed.

Degradation Strategy

Combine distributed locks with unique indexes or optimistic locks. Pure distributed locks may expire, and backend services might retry after timeout, so a secondary uniqueness check remains essential.

Conclusion

The article introduced duplicate request scenarios, defined idempotency, and detailed common backend solutions, then demonstrated their application in an order process. These guidelines help design robust anti‑repeat and idempotent controls.