How Apache Pulsar Handles Delayed Message Delivery and Its Scaling Challenges

What is delayed message delivery

In message‑queue systems a delayed message is stored on the broker and not delivered to the consumer until a specified time has passed. Two variants are common:

Scheduled (timed) messages : the producer provides an absolute future timestamp.

Delayed messages : the producer provides a relative delay interval.

Typical use cases

Retrying failed service requests after a back‑off period.

Sending payment reminders and automatically closing unpaid orders after a timeout.

Sending reminder notifications shortly before scheduled interviews or meetings.

Using Pulsar delayed messages

Pulsar added delayed delivery in version 2.4.0. Producers can specify the delay with deliverAfter (relative) or deliverAt (absolute). The client computes the target timestamp and sends it to the broker.

producer.newMessage()
    .deliverAfter(long time, TimeUnit unit)
    .send();

producer.newMessage()
    .deliverAt(long timestamp)
    .send();

Pulsar supports delays ranging from seconds to months and allows mixing delayed and non‑delayed messages in the same topic.

Implementation principle

When a delayed message arrives, Pulsar stores it in a temporary location and creates an index composed of timestamp|LedgerID|EntryID. The Delayed Message Tracker maintains an in‑memory priority queue sorted by the timestamp. Consumers first query the tracker; if a message’s delay has expired, the tracker returns the index so the broker can deliver the message. If no delayed messages are due, normal messages are consumed. If a broker crashes or a topic’s ownership moves, Pulsar rebuilds the delayed index queue to preserve continuity.

Challenges at large scale

Memory consumption of the delayed index queue : each delayed message adds three long values to the index. The index is maintained per subscription, so many subscriptions or long‑duration delays can exhaust memory.

Rebuilding the index after failures : when a broker restarts or ownership transfers, reconstructing the index for large volumes with long delays can take hours, even with partition‑level parallelism.

Future work

The community plans to add time‑based partitioning to the delayed index. Only the nearest time slice (e.g., the next five minutes) would stay in memory, while older slices are persisted to disk. This reduces both memory pressure and rebuild latency.

Conclusion

Apache Pulsar provides second‑level precision delayed delivery via deliverAfter and deliverAt. The design is efficient for moderate workloads, but scaling to massive delayed queues requires addressing memory limits and index‑rebuild latency, motivating the proposed time‑partitioned index approach.