Message Duplication Is Inevitable: Building a Multi‑Layer Idempotency Middleware for Ten‑Million QPS

Incident Overview: Duplicate Coupon Distribution

During a midnight promotion, users received three identical $200‑minus‑$50 coupons despite a limit of one per user. Investigation traced the issue to a Kafka broker restart that re‑delivered uncommitted messages, and a previously unique index that had been unintentionally changed to a regular index.

Why Duplicates Are Inherent to Message Queues

Most mainstream queues adopt an "at‑least‑once" guarantee: producers resend if they lack an ack, and brokers resend if consumers haven’t committed offsets. Each retry can generate a duplicate, so duplication is a feature, not a bug.

Challenges of Business‑Level Idempotency

Teams often implement idempotency themselves using various techniques—database unique constraints, optimistic‑lock version fields, Redis SETNX, or in‑memory maps. In large microservice landscapes this leads to inconsistent implementations, hidden edge‑cases, and increased operational burden.

Implementation diversity : different services choose different mechanisms, causing divergent bugs.

Boundary conditions : "check‑then‑write" patterns can race under load, exposing duplicate processing only in production.

Idempotent key definition : using messageId fails because it changes on retries; business fields may also collide.

Lack of observability : custom idempotency logic rarely has unified metrics, making it hard to know how many duplicates were intercepted.

Typical Business‑Level Idempotency Patterns

Unique Index

Leverage a database unique constraint (e.g., user_id + product_id + batch_id) so duplicate inserts raise a primary‑key conflict, which the application treats as "already processed". Strong consistency but requires schema changes.

State Machine

Define a strict progression of states (e.g., "pending" → "paid" → "shipped" → "completed"); if a message arrives when the state is already beyond the expected step, it is ignored. Works well for well‑structured workflows.

Optimistic‑Lock Version

Maintain a version column; an update succeeds only if the version matches, otherwise the row count is zero, indicating a duplicate.

流水表 + Unique Constraint

Create a dedicated "flow" table with a unique key for each message; insertion succeeds only once, providing a lightweight idempotency layer that minimally impacts existing tables.

System‑Level Idempotency Middleware

The mature approach extracts the "has this message been processed" check from business code and implements it in a shared middleware. The middleware handles key generation, storage, and TTL, while business services only need to declare the idempotent key.

Idempotent‑Key Design Principles

Business‑agnostic : generated by the producer or extracted automatically, not crafted in business logic.

Deterministic across retries : the same key must be produced for a given logical message, ruling out volatile messageId.

Globally unique : combine business identifier, primary key, and a time window (e.g., bizId = userId + activityId + batchId).

Deduplication Storage Options

Relational DB table : strong consistency, but limited throughput at tens of millions of writes per second.

Redis : fast SETNX or SET NX EX, but memory‑bound and vulnerable to data loss on failover.

Bloom filter : memory‑efficient pre‑filter with false positives, unsuitable as the sole source of truth.

Distributed KV (HBase, TiKV) : high QPS scalability, weaker consistency, higher ops overhead.

There is no perfect storage; the choice depends on scale: ~100k QPS can use MySQL, ~1M QPS often moves to Redis, and ~10M QPS typically combines Bloom filter, Redis cluster, and persistent storage.

Challenges at Ten‑Million QPS

Storage Capacity Explosion

Assuming a 64‑byte key, 10 M messages per second generate 640 MB/s, or ~55 TB per day. Even with short TTLs, storage pressure is significant.

Hotspot on Single Idempotent Key

Massive promotions cause many users to request the same coupon, concentrating load on a single Redis shard. Sharding strategies must align with access patterns (user‑based vs. message‑based).

Cross‑Datacenter Consistency

Multi‑datacenter deployments can cause the same message to be processed in different regions; a global deduplication service is needed, though it may become a bottleneck.

Long‑Term Idempotent‑Key Expiration

TTL‑based keys expire after hours; messages delayed for days (e.g., offline compensation) would be treated as new. The solution is to separate "system‑level" (short‑window) idempotency from "business‑level" (long‑window) checks using persistent unique indexes.

Layered Defense Architecture

The recommended architecture stacks multiple defenses:

Production side : embed a deterministic idempotent key in the message payload.

Middleware layer : on consumption, check Redis (or equivalent) for the key; if present, skip processing; set a short TTL (e.g., 1 hour).

Business table layer : retain a unique index on the core table as the final safeguard.

Status layer : use optimistic locking for counters.

Reconciliation layer : nightly jobs compare emitted coupons with recorded entries and alert on mismatches.

If any layer fails, the next one catches the duplicate, achieving near‑100 % reliability without relying on a single perfect solution.

Evolution Stages

Four maturity stages are described:

Business autonomy : each team implements its own idempotency; suitable below 100k QPS.

Half‑infrastructure : a shared SDK or JAR provides utilities; handles up to ~1M QPS.

Platform stage : idempotency becomes a built‑in middleware capability with unified monitoring and governance; required for ~10M QPS.

Intelligent stage : dynamic scaling, automatic storage tiering, and predictive caching are explored.

The article concludes with two open questions about middleware failure handling and responsibility boundaries in multi‑active architectures, emphasizing that understanding trade‑offs is more valuable than memorizing any single solution.