Idempotency Strategies for APIs: Preventing Duplicate Submissions

Network timeouts, client retries, user double‑clicks, and at‑least‑once MQ delivery can cause the same intent to reach the server multiple times. Idempotency guarantees that executing the request many times yields the same business result as a single successful execution.

1. What is idempotency and what problem it solves

Repeated business requests should have side effects only once, and subsequent calls can safely return the first (or equivalent) result.

In HTTP, GET, PUT, DELETE are usually idempotent, while POST is not. The real trouble is with “create‑type” write operations such as order placement, payment callbacks, deduction, coupon issuance, inventory reservation.

User double‑click / weak‑network retry: two orders from one cart, or a coupon redeemed twice.

Caller timeout retry: the request succeeded but the response was lost, causing a second write.

MQ at‑least‑once: broker re‑pushes or consumer times out and re‑processes the same message.

Idempotency does not try to block requests; it allows repeated arrivals while keeping the result predictable. It is adjacent to rate‑limiting and distributed locks but serves a different purpose: limiting duplicate execution of the same intent.

2. Common implementation methods and applicable scenarios

In production rarely a single technique is enough. The following are presented from light to heavy: natural business idempotency → storage constraints → token deduplication → state machine.

2.1 Natural business idempotency / conditional update

Design write operations as “set to target state” instead of “increment once”. Examples:

Mark an order as paid:

UPDATE orders SET status='PAID' WHERE id=? AND status='WAIT_PAY'

Inventory release based on a voucher: if the voucher does not exist or is already released, the operation succeeds directly.

When the same call is repeated, the affected row count is 0, which the business treats as success. Suitable for domains with clear state transitions (payment success, cancellation, ticket issuance). Low implementation cost but requires the domain model to be re‑entrant.

2.2 Database unique constraints

Create a unique index on a “business unique key”. Insert collisions are treated as duplicate requests. Typical keys:

Order deduplication: (user_id, client_request_id) or order number

Payment callback: payment_id / out_trade_no MQ consumption: message_id in a deduplication table

First: INSERT succeeds → continue business
Duplicate: unique‑key conflict → fetch existing result (or ACK)

Suitable for creation‑type interfaces where a stable business key can be extracted. The conflict branch must retrieve the first result and return it; otherwise the client may keep retrying.

2.3 Idempotency Token (Idempotency‑Key)

The client obtains (or generates) a one‑time token and sends it in a header or parameter; the server records the token in Redis or a database together with the processing result.

Suitable for open APIs, payment gateways, front‑end duplicate‑submission prevention—callers can reliably hold the same key (UUID or server‑issued). Points to watch:

The key should contain business dimensions (e.g., user ID) to avoid cross‑user collisions.

Record should have TTL and distinguish “processing / success / failure”. Whether a failed key can be retried must be defined by product.

Pure Redis deduplication may lose records on crash; for loss‑critical paths store in DB or write a deduplication table within the same transaction.

2.4 State machine + optimistic lock (version / CAS)

Applicable to long‑lived objects with multiple entry points that modify the same aggregate (e.g., orders that receive payment callbacks, timeout cancellations, user cancellations concurrently). Complements unique constraints: unique constraint prevents extra rows, state machine prevents a row from entering the wrong stage.

Two layers:

State machine can provide business idempotency; optimistic lock itself is not idempotent.

State‑machine / conditional update ensures idempotency: repeated calls on the same resource produce unchanged results. Example for payment success:

-- First: WAIT_PAY → PAID, affects 1 row
UPDATE orders SET status='PAID' WHERE id=? AND status='WAIT_PAY';
-- Duplicate: already PAID, affects 0 rows → treat as success

Version / CAS mainly prevents concurrent overwrites; it does not automatically make the operation idempotent. Update pattern: WHERE id=? AND version=? If a conflict occurs and the server simply returns an error, the caller will retry and the operation is not idempotent. Whether it counts as idempotent depends on how illegal states or conflicts are handled.

Already in target state → success (idempotent)

Illegal state (e.g., pay after cancel) → business error (generally not idempotent)

Only version conflict → failure / client retry (not idempotent, concurrency control)

In practice, combine a condition on status (e.g., AND status='WAIT_PAY') for re‑entrancy and use version as a concurrency aid. Locks / optimistic locks mitigate race windows, but idempotency still relies on “same intent repeated, business result unchanged”.

2.5 Distributed lock (auxiliary, not core idempotency)

Acquire a short‑lived lock on the same key, serialize concurrent requests, then perform “check‑then‑write”. Locks only shrink the race window and cannot replace unique constraints or state machines—if the lock crashes or TTL expires, double writes may still happen. Suitable for “check‑then‑write” scenarios that need quick mitigation; long‑term solution should fall back to storage‑level constraints.

3. Points to watch

Define clearly what a duplicate request returns: replay the first successful response (including orderId) or return an empty success indicating “already processed”. Callers decide whether to retry based on this.

Token must bind to the intent, not to each click: generate and cache the key when entering the checkout page or opening a confirmation dialog; reuse it for retries and clear after success. Generating a new UUID on every click would make double clicks legitimate separate requests.

Loss‑critical paths should layer two protections: front‑end token improves experience, while DB unique constraint + state machine provide a fallback. Relying solely on Redis deduplication risks double writes on crash.

Monitor: unique‑key conflict count, token “processing” timeouts, deduplication table growth; these metrics reveal front‑end key misuse or downstream retry storms early.

Conclusion

Idempotency is not about forbidding retries; it is about making retries safe: the same intent arriving multiple times should produce the same business result as a single successful execution.

Choosing a solution by scenario:

State change (payment, cancel): conditional update / state machine; treat target state as success.

Document creation (order, callback): business unique key + DB unique constraint.

Front‑end double‑click / open API: Idempotency‑Key, reusing the same key for the same intent.

Message consumption: design for at‑least‑once delivery, deduplicate or make business logic re‑entrant.

Locks and version numbers only alleviate concurrency conflicts; they cannot serve as idempotency on their own. Effective idempotency relies on preventing side effects on repeated execution and letting storage constraints catch double writes.