Preventing Duplicate Consumption in Kafka: Design, Idempotence, and Configuration Strategies

Message and Business Design

Define a globally unique identifier (e.g., UUID or business primary key) in each Kafka message. Include metadata such as timestamp, source system, and schema version. This identifier is used for idempotence checks and tracing. Choose the delivery guarantee (at‑least‑once or exactly‑once) that matches business semantics and design the processing flow accordingly.

Consumer‑Side Idempotence and Deduplication

Implement idempotent consumption by persisting a deduplication record for each message identifier. Typical approaches:

Database table with a unique constraint on the identifier.

Distributed cache (e.g., Redis, Hazelcast) that supports atomic “set‑if‑absent” operations.

Processing sequence:

Read the message.

Attempt to insert the identifier into the deduplication store.

If the insert succeeds, execute business logic.

Commit the consumer offset only after the business transaction completes.

If the storage system supports transactions, combine the business update and offset commit in a single transaction to eliminate race conditions.

Kafka Configuration and Feature Utilization

Key Kafka settings to reduce duplicate consumption:

Disable enable.auto.commit and perform manual offset commits after successful processing.

Use a transactional producer ( transactional.id) to achieve end‑to‑end exactly‑once semantics.

Allocate an appropriate number of partitions and configure consumer concurrency so that each partition is processed by a single thread, avoiding concurrent commits on the same partition.

Define retry policies (e.g., retries, retry.backoff.ms) and a dead‑letter queue (DLQ) topic for messages that repeatedly fail.

System‑Level Guarantees and Monitoring

In high‑throughput environments, additional safeguards are recommended:

Enable the idempotent producer flag ( enable.idempotence=true) to prevent duplicate writes from the producer side.

Use distributed locks or optimistic concurrency control when multiple services need to update shared state.

Instrument metrics for duplicate‑consumption rate, offset‑commit latency, and consumer lag; set up alerts to detect anomalies early.

When a duplicate is detected, use replay or rollback mechanisms based on the persisted identifier.