How Apache Pulsar Guarantees No Duplicate Delivery: A Deep Dive into Ack Modes

Why Message Acknowledgment Matters

In Apache Pulsar, a consumer must acknowledge a message after processing it; only then does the broker consider the message fully consumed and stop redelivering it. Proper acknowledgment prevents duplicate deliveries and ensures reliable consumption.

Acknowledgment Modes in Pulsar

Pulsar provides four distinct acknowledgment mechanisms:

Single Message Acknowledge – acknowledges an individual message.

Cumulative Acknowledge ( AcknowledgeCumulative ) – acknowledges a message and all previous messages in the same subscription.

Batch‑Level Single Acknowledge – acknowledges selected messages inside a batch when the broker configuration AcknowledgmentAtBatchIndexLevelEnabled is turned on.

Negative Acknowledge – tells the broker to redeliver the message later.

Supported Ack Scenarios per Subscription Type

The following subscription modes have different capabilities:

Exclusive : supports single, cumulative, and batch‑level acks.

Shared : supports single and batch‑level acks, but does not support cumulative acks.

Failover : supports all three ack types.

Key_Shared : supports single and batch‑level acks, but not cumulative acks.

Implementation of Acknowledge and AcknowledgeCumulative

Both methods delegate the request to an AcknowledgmentsGroupingTracker. Pulsar has two implementations: one for persistent subscriptions ( PersistentAcknowledgmentsGroupingTracker) and a no‑op version for non‑persistent subscriptions. The tracker batches acknowledgments to avoid overwhelming the broker. By default, acknowledgments are collected for up to 100 ms or until 1 000 pending acks accumulate, then sent as a batch. The client can disable batching by setting AcknowledgementGroupTimeMicros to 0, causing an immediate request for each ack.

Batch Message Acknowledgment

For batch messages, Pulsar uses a PendingIndividualBatchIndexAcks map where the key is the batch MessageId and the value is a BitSet indicating which entries in the batch are acknowledged. A BitSet stores each flag as a single bit, allowing 8 KB to represent the ack state of 65 536 messages, dramatically reducing memory usage.

Cumulative Acknowledgment Storage

The tracker keeps only the latest cumulative position (e.g., LedgerId=5, EntryId=10). When a newer cumulative ack arrives (e.g., 5:20), it simply replaces the previous position.

Flush Mechanism

All pending acknowledgments are eventually sent to the broker when the tracker’s flush method is invoked. The same command is used for every ack type; the only difference is the AckType field that indicates whether it is a single, cumulative, batch‑level, or negative ack.

Negative Acknowledgment

Negative acks are processed by a NegativeAcksTracker. The tracker records each message together with a delay (default 1 minute) and uses Pulsar’s internal time‑wheel (≈33 ms tick) to check when the delay expires. Expired messages trigger a RedeliverUnacknowledgedMessages command to the broker. The tracker also merges multiple negative acks to reduce request volume.

Handling Unacknowledged Messages

Two scenarios are considered:

If a consumer has already called receive() (or the async callback) but never acked the message, the message is tracked by UnAckedMessageTracker. This tracker maintains a time‑wheel based on AckTimeout and TickDurationInMs. When a slot expires, the tracker automatically triggers redelivery.

If messages are pre‑fetched but the application never calls receive(), they remain in the local queue. The size of this pre‑fetch queue is controlled by the ReceiveQueueSize parameter. These messages are marked as PendingAck on the broker side and will not be redelivered unless the consumer is closed.

Both mechanisms ensure that messages are eventually either acked or redelivered, preventing indefinite loss.

Operational Tips

When creating a consumer, choose ReceiveQueueSize carefully to avoid excessive local buffering. Enabling ConsumerStatsRecorder lets the client periodically emit metrics such as pending message count and total received messages, which helps diagnose performance issues.