Master Kafka’s Delivery Guarantees: At‑Most, At‑Least, and Exactly‑Once Explained

Kafka Delivery Semantics

Kafka provides three possible message delivery guarantees: At most once (messages may be lost but are never redelivered), At least once (messages are never lost but may be redelivered), and Exactly once (each message is delivered only once).

Producer Perspective

At most once : the producer sends a message and does not wait for any acknowledgment, so loss is possible but duplication cannot occur.

At least once : the producer waits for an acknowledgment; if the ack is not received it retries, which can cause duplicate messages.

Exactly once : the producer operates idempotently, ensuring that repeated sends result in a single stored record.

Producer Configuration

At least once (default) : no special configuration; Kafka defaults to acks=1 and retries=2147483647.

At most once : set acks=0 (no broker acknowledgment) and optionally retries=0. Be aware that max.in.flight.requests.per.connection must be 1 to avoid out‑of‑order delivery.

Exactly once : enable idempotence with enable.idempotence=true and set acks=all; optionally keep max.in.flight.requests.per.connection below 5.

Kafka’s default at‑least‑once semantics stem from acks=1 and a very high retries value.

How Kafka Achieves Idempotence

Each producer receives a unique PID and a monotonically increasing sequence number for every <topic, partition>. The broker stores the highest sequence number per PID; messages with a lower or equal sequence are discarded, guaranteeing exactly‑once delivery per partition.

Properties props = new Properties();
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put("acks", "all"); // default when idempotence is true
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer producer = new KafkaProducer(props);
producer.send(new ProducerRecord(topic, "test"));

Consumer Perspective

At most once : the consumer may commit the offset before processing; if processing fails, the message is lost.

At least once : the consumer processes the message first and commits the offset afterwards; failures cause the same message to be re‑processed.

Exactly once : processing and offset commit are atomic, typically achieved with Kafka transactions and isolation.level=read_committed.

Consumer Configuration

At least once : set enable.auto.commit=false and manually call consumer.commitSync() after successful processing.

At most once : keep enable.auto.commit=true and set a very small auto.commit.interval.ms so offsets are committed before processing.

Exactly once : set isolation.level=read_committed and use transactional producers.

Preventing Data Loss

Loss can occur at the producer, broker, or consumer level. Mitigations include:

Producer retries and callbacks; use acks=all to require all replicas to acknowledge.

Broker replication factor ≥ 3 and min.insync.replicas>1 to ensure writes are persisted to multiple brokers.

Disable unclean leader election ( unclean.leader.election.enable=false) to avoid electing out‑of‑sync replicas.

Message Deduplication and Transactions

Kafka’s idempotent producer removes duplicate records on the broker side. For cross‑partition or cross‑session guarantees, use transactional producers: enable idempotence, set a transactional.id, and wrap sends between beginTransaction() and commitTransaction(). Consumers must read with isolation.level=read_committed to see only committed data.

producer.initTransactions();
try {
    producer.beginTransaction();
    producer.send(record1);
    producer.send(record2);
    producer.commitTransaction();
} catch (KafkaException e) {
    producer.abortTransaction();
}

Message Ordering

Kafka guarantees order only within a partition. Out‑of‑order delivery can happen when retries are performed; setting max.in.flight.requests.per.connection=1 prevents this at the cost of throughput.

Images