Ensuring Idempotency in Distributed Systems: Strategies and Best Practices

Background

Many real‑world operations must produce the same effect and return the same result no matter how many times they are executed. Examples include preventing duplicate front‑end submissions, ensuring a payment is deducted only once, sending a single SMS, and creating only one business order.

Idempotency Concept

Idempotency (idempotent, idempotence) originates from mathematics and abstract algebra. In programming, an idempotent operation yields the same impact whether executed once or many times; an idempotent function returns the same result with identical parameters and does not alter system state.

Technical Solutions

Query Operations – SELECT statements are naturally idempotent; repeated queries return identical results when data is unchanged.

Delete Operations – Deleting the same record multiple times has the same effect (the record is removed). Note that the returned count may differ (0 if already absent).

Unique Indexes – Adding a unique or composite index prevents duplicate dirty data, e.g., a user can have only one financial account; the database enforces uniqueness.

Token Mechanism – Before submitting data, request a token stored in Redis or JVM memory with an expiration time. After submission, the backend validates and deletes the token, then issues a new one. This prevents duplicate submissions caused by repeated clicks, network retries, or proxy resends. Use Redis DELETE for atomic validation; avoid SELECT + DELETE due to concurrency issues.

Pessimistic Lock – Acquire a row lock with SELECT * FROM table_xxx WHERE id='xxx' FOR UPDATE. The locked row must be identified by a primary key or unique index to avoid table‑wide locking.

Optimistic Lock – Update only when a version or condition matches. Example using a version column:

update table_xxx set name=#name#, version=version+1 where version=#version#

Or using a conditional check to ensure sufficient balance:

update tablexxx set avaiamount=avaiamount-#subAmount# where avaiamount-#subAmount# >= 0

Use primary key or unique index in the WHERE clause to avoid full‑table locks.

Distributed Lock – In a distributed system, obtain a lock from a third‑party service (Redis, Zookeeper) before inserting or updating data, then release it after the operation.

Select + Insert Pattern – For low‑concurrency jobs, first query to see if the work has already been done, then proceed only if not processed.

State‑Machine Idempotency – Design business processes as finite state machines; if a request tries to move to a previous state that has already been passed, reject it, ensuring idempotent state transitions.

API Idempotency for External Calls – Require callers to provide a source identifier and a sequence number (source+seq). Create a unique composite index on these fields to prevent duplicate processing of payment requests or other critical operations.

Conclusion

Idempotency is a fundamental trait of a competent programmer. When designing systems—especially those handling financial transactions—ensuring operations are idempotent guarantees efficiency, data accuracy, and a better user experience by preventing issues such as double charges or duplicate messages.