Mastering Kafka Idempotent Producers and Transactional Messaging

Kafka is a core component of large‑scale architectures; this article explains how to implement an idempotent Kafka producer.

Enabling Idempotent Producer

Set enable.idempotence=true on the producer side. The essential configuration properties are:

properties.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); // core switch
properties.put(ProducerConfig.ACKS_CONFIG, "all"); // must be all for durability
properties.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE); // maximum retries
properties.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5); // max 5 to preserve order

The producer maintains a Producer ID (PID) and an incrementing sequence number for each session. The broker uses the PID and sequence number to detect and discard duplicate writes to the same partition, guaranteeing “at most once” delivery within a single producer session.

Advantages: transparent, low latency, no extra storage.

Limitations: works only within a single producer session, suitable for most single‑node or single‑session idempotent scenarios.

Transactions

While idempotence ensures per‑partition idempotency, it does not address cross‑partition or cross‑session idempotency and atomicity. To achieve:

Cross‑partition idempotency

Atomic writes across multiple topics

Exactly‑once semantics

you need to use Kafka transactions. Transactions introduce additional complexity, performance overhead (transaction coordinator involvement), and require retry handling for conflicts.

External Deduplication (Application‑Level Idempotence)

When built‑in idempotence or transactions cannot meet business requirements, an external deduplication mechanism can be added at the application layer:

Define a business‑unique key (e.g., order ID, request ID) and embed it in the message payload.

Use external storage (such as Redis or a database) to check for the key before processing.

This approach is flexible, version‑agnostic, and can handle complex cross‑process or cross‑service duplicate scenarios, but it adds storage requirements and consistency design, potentially increasing latency and operational cost.