Preventing Kafka Duplicate Consumption with Idempotent Design

Business Idempotency (Recommended)

The most stable and controllable solution is to ensure each message carries a globally unique business ID (e.g., order number, payment transaction ID). Before processing, the consumer checks whether this ID has already been handled.

Implementation approaches:

Database deduplication table : Create a table such as (id PK, consume_time, business_status, ...). Inserting uses a primary key or unique index so the same ID can succeed only once; subsequent attempts fail and are skipped. Suitable for moderate QPS scenarios that require strong consistency.

Redis/cache deduplication : Use SETNX or a GET+SET pattern with the message ID as the key. Write succeeds only once, optionally with an expiration to limit storage. Fits high‑throughput cases where slight inconsistency is tolerable.

Idempotent Producer and Consumer

Enable idempotence on the Kafka producer by setting enable.idempotence=true, which prevents duplicate writes caused by retries.

On the consumer side, implement idempotent processing by checking the message's unique identifier (key, business ID, or hash) against external persistent storage (database unique index or dedup table) or a cache (e.g., Redis SETNX). Ensure the deduplication check and the subsequent business write are performed atomically, using transactions or other atomic operations to avoid race conditions.

Exactly-Once Semantics

Kafka provides a transaction mechanism that, combined with an idempotent producer, can guarantee atomic commits across multiple partitions or topics, achieving end‑to‑end exactly‑once delivery.

In stream processing frameworks such as Kafka Streams or any consumer implementation that supports transactions, offset commits can be included in the same transaction, ensuring that both data writes and offset advances are committed together.

This approach is suitable for scenarios demanding high consistency and where the added complexity and performance overhead are acceptable.