Ensuring Message Reliability, Idempotence, and Transactions in Kafka

Reliability

Kafka only guarantees persistence for messages that have been committed , i.e., when a configurable number of broker replicas have written the record to their log.

How to Ensure No Message Loss

Committed messages are those that all required replicas have stored; if at least one replica among the N replicas remains alive, the message is not lost.

Data‑Loss Scenarios

Producer‑Side Loss

Because the Kafka producer sends asynchronously, a call to producer.send(msg) does not guarantee delivery; therefore the producer should always use the callback‑based API producer.send(msg, callback) to handle failures.

Consumer‑Side Loss

If a consumer updates its offset before processing the record and crashes, the record is lost; the correct order is to process the record first and then commit the offset, which may cause duplicate consumption.

When a consumer processes records in multiple asynchronous threads while auto‑committing offsets, a thread failure can also cause loss; disabling auto‑commit and committing offsets manually avoids this.

Best Practices

Use producer.send(msg, callback).

Set acks=all so that all replicas must acknowledge a write before it is considered committed.

Configure a large retries value to let the producer automatically retry transient failures.

Set unclean.leader.election.enable=false to prevent out‑of‑sync brokers from becoming leaders.

Use replication.factor>=3 for sufficient redundancy.

Set min.insync.replicas>1 so a message is considered committed only after being written to at least two replicas.

Ensure replication.factor > min.insync.replicas; a common rule is replication.factor = min.insync.replicas + 1.

Disable automatic offset commits ( enable.auto.commit=false) and commit offsets only after processing succeeds.

Explanation of items 2 and 6: if the ISR contains only one replica, acks=all behaves like acks=1; min.insync.replicas adds a lower bound to the number of replicas that must acknowledge a write.

Idempotence

Since version 0.11.0.0, Kafka provides an idempotent producer that can be enabled with enable.idempotence=true (or ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG=true), allowing the broker to deduplicate messages.

Scope: it guarantees idempotence only per partition and only for the lifetime of a single producer instance; after a restart the guarantee does not hold.

Transactions

Kafka 0.11 introduced transactional support with a read_committed isolation level, ensuring that multiple records are written atomically to target partitions and that consumers see only committed transactions.

Transactional Producer

To produce atomic writes across partitions, enable idempotence, set a transactional.id, and use the transaction API:

producer.initTransactions();
try {
    producer.beginTransaction();
    producer.send(record1);
    producer.send(record2);
    producer.commitTransaction();
} catch (KafkaException e) {
    producer.abortTransaction();
}

This code guarantees that record1 and record2 are committed as a single transaction.

Consumer Configuration

Set the isolation.level parameter to either read_uncommitted (default, sees all messages) or read_committed (sees only messages from committed transactions; non‑transactional messages are always visible).