Duplicate Requests Aren’t Bugs: 5 Idempotency Solutions for Distributed Systems

Problem illustration

During a payment, the user clicks "Pay" and the network times out. The front‑end automatically retries, the request reaches the server and the charge succeeds, but the response is lost. The user clicks again, causing a second charge, and after seeing an error clicks a third time, resulting in a third charge. This shows that duplicate requests are not rare bugs but an inevitable condition in distributed systems, caused by network retries, at‑least‑once message delivery, user double‑clicks, and micro‑service retry policies.

Idempotency definition

Idempotency means that executing the same operation once or many times yields exactly the same effect (mathematically, f(f(x)) = f(x)). In HTTP, RFC 7231 defines GET, HEAD, PUT and DELETE as idempotent, while POST is not because it usually creates a new resource. Idempotent does not equal safe (no side‑effects) and does not mean the operation does nothing; it means repeated execution does not produce additional impact.

Sources of duplicate requests

Network retries – HTTP clients and gateways (e.g., Nginx proxy_next_upstream, Feign retryer) retry when a response is lost.

Message re‑consumption – At‑least‑once delivery semantics of queues (Kafka, RabbitMQ) cause a consumer to process the same message after a restart.

User double‑click – Front‑ends without debounce or button disabling let users submit the same request multiple times.

Micro‑service retries – Service‑to‑service retry policies (Spring Retry, Resilience4j) may resend a request after the downstream service has already processed it.

Five implementation schemes

1. Database unique index

Use a unique constraint on a business key (e.g., order_id). Duplicate INSERTs are rejected by the database.

-- Insert order with unique index
INSERT IGNORE INTO orders (order_id, user_id, amount)
VALUES ('ORD001', 'U001', 100.00);
-- Or update on duplicate
INSERT INTO orders (order_id, user_id, amount)
VALUES ('ORD001', 'U001', 100.00)
ON DUPLICATE KEY UPDATE amount = amount;

Pros: Simple, relies on ACID guarantees, high reliability.

Cons: Only prevents duplicate inserts, not duplicate updates; can become a performance bottleneck under high concurrency; sharding requires globally unique IDs.

Applicable scenarios: Pure creation operations such as registration, order creation, coupon issuance.

2. Anti‑replay token

Generate a one‑time token, store it in Redis with a TTL, and delete it atomically when the request is processed.

// 1. Get token
token = UUID.randomUUID()
SETNX "token:" + token, "1", TTL=600s // store in Redis

// 2. Submit request
result = DEL "token:" + token // atomic delete
if result == 0:
    return "duplicate request"
// proceed with business logic

Pros: Low intrusion, works for any API, no DB dependency.

Cons: Requires an extra request to obtain the token; token deletion and business execution are not atomic unless wrapped in a Lua script; Redis failure can lose tokens.

Applicable scenarios: Form submissions, payment interfaces where explicit anti‑replay is needed.

3. State machine / optimistic lock

Leverage business state transitions or version numbers. A request is accepted only when the current state/version matches the expected value.

-- State machine: only UNPAID can become PAID
UPDATE orders SET status = 'PAID'
WHERE order_id = 'ORD001' AND status = 'UNPAID';

-- Optimistic lock: version must match
UPDATE orders SET amount = 200.00, version = version + 1
WHERE order_id = 'ORD001' AND version = 3;

Pros: Naturally fits business semantics; optimistic lock offers good concurrency for read‑heavy workloads.

Cons: Requires explicit state definition; unsuitable for stateless operations; optimistic lock may suffer high conflict rates under extreme concurrency.

Applicable scenarios: Order status flows, approval processes with clear state transitions.

4. Redis SETNX deduplication

Each request carries a unique ID. Use Redis SETNX (set if not exists) to atomically claim the ID.

// Request carries requestId
result = SETNX "req:" + requestId, "1", TTL=3600s
if result == 0:
    return "duplicate request"
// execute business logic

Pros: High performance, simple implementation, automatic TTL cleanup.

Cons: Depends on Redis availability; SETNX success does not guarantee business success, leading to the "SETNX + business failure" pitfall unless wrapped in a Lua script.

Applicable scenarios: API deduplication, message‑consumer deduplication.

5. Downstream deduplication table

Create a dedicated table that records processed request IDs. Insert the record and perform the business update in the same database transaction.

BEGIN TRANSACTION
  INSERT INTO idempotent_log (request_id, result) VALUES ('REQ001', '...');
  UPDATE account SET balance = balance - 100 WHERE user_id = 'U001';
COMMIT
-- If request_id already exists, INSERT fails and the whole transaction rolls back

Pros: Strongest consistency—both dedup record and business change succeed or fail together; no external middleware.

Cons: Table size grows continuously; requires periodic cleanup; adds write load to the primary database.

Applicable scenarios: Core financial or trading systems where any discrepancy is unacceptable.

Comparison and selection guidance

State‑flow present: Prefer state machine or optimistic lock.

Pure insert use‑case: Use a unique index.

Need a generic anti‑replay layer: Choose token or Redis SETNX.

Extreme consistency requirement: Use a downstream deduplication table within a transaction.

In practice, teams often combine schemes: a fast SETNX layer to filter most duplicates, followed by a state‑machine or unique‑index guard for business‑level safety.

Practical details often overlooked

Idempotency key design: Choose dimensions that uniquely identify the operation (e.g., user ID + action + business ID). Wrong dimensions lead to false positives.

Expiration & cleanup: Set TTL for Redis keys; archive or purge old rows from dedup tables to avoid storage bloat.

Return value policy: On duplicate detection, return the original successful result instead of an error, so callers see consistent output.

Avoid over‑idempotent design: Not every endpoint needs idempotency; read‑only or low‑risk APIs can skip the extra guard.

Testing: Include concurrent duplicate‑request tests and message‑replay tests in CI to ensure idempotency guarantees remain intact.

Takeaways

Duplicate requests are a normal condition in distributed systems, arising from network retries, at‑least‑once messaging, user double‑clicks, and micro‑service retries.

Five concrete solutions—unique index, token, state machine/optimistic lock, Redis SETNX, and downstream dedup table—cover different business characteristics.

Real‑world deployments often layer multiple mechanisms for defense‑in‑depth.

Correct idempotency key design and proper return‑value handling are critical to a successful implementation.