How Pulsar Brokers Manage Message Acknowledgments and Cursor Gaps

Broker‑side acknowledgment management

After covering client‑side acknowledgment modes in a previous article, this piece dives into how Pulsar brokers record which messages have been consumed and prevent duplicate delivery.

Cursor fundamentals

Each subscription has a cursor that stores the current consumption position and related metadata. Persistent cursors are stored in ZooKeeper, while non‑persistent cursors reside only in broker memory. The cursor contains a Bookkeeper client reference, MarkDeletePosition (the highest entry that can be safely deleted), ReadPosition (the next entry to read), and other fields such as LastMarkDeleteEntry , CursorLedger , IndividualDeletedMessages , and BatchDeletedIndexes .

Cursor sharing and consumption modes

When multiple consumers share a subscription name, they also share a cursor. In Exclusive or FailOver modes only one consumer uses the cursor at a time, whereas in Shared or Key_Shared modes several consumers may use it concurrently, which can create acknowledgment gaps (holes).

Acknowledgment types and cursor movement

Single‑message acknowledgments move the cursor forward by one entry; cumulative acknowledgments can advance it by multiple entries. Batch acknowledgments record per‑message ack information inside BatchDeletedIndexes. Negative acknowledgments do not affect the cursor and are omitted from this discussion.

Gap (hole) management

In shared consumption, unacknowledged entries create gaps. Pulsar stores these gaps using Guava Range objects, e.g. [(1:-1,1:2],(1:3,1:6]], which efficiently represent continuous acked ranges. When the cursor’s MarkDeletePosition reaches a point where all preceding entries are acked, it can advance.

Storage of gap information

The gap container IndividualDeletedMessages is implemented as a LongPairRangeSet. The default implementation wraps Guava Range, while an optimized version ConcurrentOpenLongPairRangeSet uses a BitSet to reduce memory usage when many small gaps exist.

Persistence strategy

Cursor metadata is persisted to ZooKeeper at three moments: when the cursor is closed, when a ledger switch changes the cursorLedger, and when persisting gap data to BookKeeper fails. ZooKeeper holds only index information (cursor ledger name/ID, LastMarkDeleteEntry, and last activity timestamp), while the bulk data resides in BookKeeper.

Recovery process

On broker restart, ZooKeeper acts as a checkpoint: the broker reads the cursor index, retrieves the corresponding BookKeeper ledger, and restores the latest MarkDeletePosition and gap data.

Configuration knobs

Enable the optimized BitSet‑based RangeSet: managedLedgerUnackedRangesOpenCacheSetEnabled=true Limit the amount of gap data persisted to ZooKeeper:

managedLedgerMaxUnackedRangesToPersistInZooKeeper=1000

Protobuf definition of cursor state

message PositionInfo {
   required int64 ledgerId = 1;
   required int64 entryId = 2;
   repeated MessageRange individualDeletedMessages = 3;
   repeated LongProperty properties = 4;
   repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 5;
}

Scalability challenges and upcoming improvements

When many subscriptions generate large numbers of gaps, memory consumption grows and a single ledger entry may exceed the 5 MB size limit. A pending Pulsar Improvement Proposal (PIP) introduces an LRU‑plus‑segmented storage design: hot gap ranges stay in memory, cold ranges are evicted, and gaps are split across multiple entries with a special Marker entry that records the index of all split parts. This approach reduces memory pressure and avoids entry‑size limits.

The broker’s ManagedLedger uses a LinkedHashMap to implement an LRU list. A background thread checks memory usage; when a threshold is reached, the least‑used ledger data is swapped out. On a cache miss, the marker is consulted to reload the required entries synchronously.

Conclusion

Pulsar’s cursor and gap management involve a combination of ZooKeeper indexing, BookKeeper persistence, range‑set data structures, and LRU‑based memory control. Ongoing work aims to further optimise storage and reduce the impact of massive subscription counts.