Master Kafka’s Message Delivery Semantics: At-Most-Once, At-Least-Once, Exactly-Once

Background

After reading the Kafka official documentation on message delivery semantics, the author found the material abstract and created a simplified summary with plain language examples and a concluding overview.

What levels of message delivery semantics exist?

Message delivery guarantees refer to how messages are transferred among the producer, broker, and consumer.

At-most-once – messages may be lost but are never duplicated. At-least-once – messages are never lost but may be duplicated. Exactly-once – each message is delivered one time and only once, which is the ideal guarantee.

Which delivery semantics does Kafka support?

The author summarizes the support based on the English documentation (see image below).

Does Kafka really support at-least-once delivery?

The answer depends on conditions; it may or may not be achieved.

Producer-side requirements for at-least-once

The producer retries sending messages when network issues occur, based on the retries setting, ensuring that a failed send is retried.

Prerequisites for the producer to achieve at-least-once:

The producer application must remain running (no restart or crash) during retries.

The network or broker must recover before the retry limit is exhausted.

Consumer-side approach

To guarantee at-least-once consumption, the consumer should manually commit offsets after processing each message. Example pseudocode:

while(true){
    consumer.poll();
    // process message
    consumer.commit();
}

Broker-side configuration

Ensuring the broker does not lose messages requires several settings:

Set producer acks to all.

Configure min.insync.replicas to at least 2.

Set replication.factor to at least 3.

Disable unclean leader election by setting unclean.leader.election.enable=false.

When the above conditions are not met, how to still achieve at-least-once

Adjust consumer and broker configurations as described, but the producer also needs persistent storage for outbound messages. If messages are persisted, a JVM restart, OOM, or crash will not cause loss, allowing at-least-once delivery once the system recovers.

For detailed explanations, see the referenced articles on Juejin.

Link: https://juejin.cn/post/7314870012975104009

(Copyright belongs to the original author, please delete if infringed.)