How to Prevent Duplicate Consumption in Kafka: Practical Strategies

What is Kafka duplicate consumption?

Kafka’s consumption semantics are “at‑least‑once”, meaning a message may be delivered to a consumer multiple times, though it will never be lost.

Impact of duplicate consumption

Repeated processing can cause various problems depending on the business logic carried by the message:

Writing the same record to a database creates duplicate rows.

Updating system state (e.g., order status, inventory) may lead to incorrect or inconsistent states.

In payment scenarios, duplicate consumption can result in multiple charges to a user.

Overall, unhandled duplicate consumption can severely affect business correctness, data reliability, and system stability.

How to handle duplicate consumption

The core principle is to make the message‑processing flow idempotent.

1. Ensure idempotency (the fundamental solution)

Implement a unique identifier mechanism:

Producer generates a globally unique ID (e.g., UUID) for each message.

Consumer records processed IDs in an external store (database, Redis, etc.). Before handling a message, it checks whether the ID has already been processed and skips the business logic if so.

IF NOT EXISTS (SELECT 1 FROM processed_log WHERE msg_id = ?)
THEN
  -- execute business logic
  INSERT INTO processed_log (msg_id) VALUES (?);
END IF;

2. Manual offset commit – reduce the chance of duplicates

Configure the consumer to disable automatic commits and commit the offset only after successful processing.

props.put("enable.auto.commit", "false");
try {
    // consume logic
    consumer.commitSync(); // commit after success
} catch (Exception e) {
    // do not commit, message will be retried
}

This precise control prevents committing offsets for failed messages, but should be combined with retry logic and a dead‑letter queue to avoid endless retries.

3. Exactly‑once processing (EOS)

Since Kafka 0.11, the transactional API together with an idempotent producer enables end‑to‑end exactly‑once semantics.

props.put("enable.idempotence", "true");
// configure transactional producer and consumer

With EOS, Kafka guarantees that each message is processed exactly once across the whole pipeline.

4. Dead‑letter queue (DLQ)

Messages that repeatedly fail (including those that cannot be made idempotent) should be sent to a DLQ. These messages can later be analyzed, diagnosed, and handled manually, preventing them from blocking normal consumption.

By applying these techniques—idempotent design, manual offset management, exactly‑once semantics, and dead‑letter queues—developers can reliably eliminate the adverse effects of Kafka duplicate consumption.