Mastering Pulsar Retention, Backlog, and TTL: Configuration Strategies and Best Practices

Overview of Pulsar Message Handling

Apache Pulsar is an Apache top‑level project that provides a cloud‑native distributed messaging platform integrating messaging, storage, and lightweight function compute. It uses a compute‑storage separation architecture, supports multi‑tenant, persistent storage, cross‑region replication, and offers strong consistency, high throughput, low latency, and high scalability.

Default Message Processing

When a consumer acknowledges a message, the broker deletes it immediately.

Unacknowledged messages are stored in the backlog.

In many production environments the default behavior is insufficient, so Pulsar offers configurable strategies to override it.

Retention Strategy

Retention allows keeping messages that have already been acknowledged. It can be configured at the namespace level either via broker.conf or the command line.

root@host:/pulsar# ./bin/pulsar-admin namespaces set-retention tenant/namespace --size 10M --time 30m

After setting retention, you can query the configuration:

$ pulsar-admin namespaces get-retention tenant/namespace
{
  "retentionTimeInMinutes": 30,
  "retentionSizeInMB": 10
}

Backlog Quota

The backlog is the collection of unacknowledged messages for a persisted topic. You can configure the maximum backlog size per namespace and define the action to take when the quota is exceeded.

root@host:/pulsar# ./bin/pulsar-admin namespaces set-backlog-quota tenant/namespace --limit 2G --policy producer_request_hold

Supported policies are producer_request_hold, producer_exception, and consumer_backlog_eviction. Query the current quota with:

$ pulsar-admin namespaces get-backlog-quotas tenant/namespace
{
  "destination_storage": {
    "limit": 2147483648,
    "policy": "producer_request_hold"
  }
}

To remove a backlog quota:

$ pulsar-admin namespaces remove-backlog-quota tenant/namespace

Clearing Backlog

When messages accumulate, you can clear the backlog. This operation is potentially dangerous, so the broker prompts for confirmation unless you use the -f (force) flag.

$ pulsar-admin namespaces clear-backlog tenant/namespace -f

Time‑To‑Live (TTL) Strategy

By default, Pulsar persists all unacknowledged messages. TTL allows you to automatically move unacknowledged messages to the acknowledged state after a configured period, after which they can be removed by the retention policy.

root@host:/pulsar# ./bin/pulsar-admin namespaces set-message-ttl tenant/namespace --messageTTL 3600

Query the TTL setting:

$ pulsar-admin namespaces get-message-ttl tenant/namespace
3600

Relationship Between TTL, Backlog, and Retention

TTL only changes the state of unacknowledged messages to acknowledged; it does not delete them. Retention governs how long acknowledged messages are kept. Backlog stores unacknowledged messages, while retention applies to acknowledged messages.

TTL vs. Dead‑Letter Queue

Both TTL and dead‑letter queues can transition unacknowledged messages to an acknowledged state. The dead‑letter queue discards the messages to a special topic, allowing you to inspect and optionally reprocess them later.

Common Pitfalls

Scenario 1: Producer sends messages with TTL enabled but no consumer is running. Since the cursor is not initialized without a consumer, TTL has no effect.

Scenario 2: Retention policy reaches its threshold but messages are not deleted. Pulsar deletes data only when the active ledger is closed and a new ledger is created; you may need to force a ledger rollover using pulsar-admin commands.

Understanding these configurations helps you fine‑tune Pulsar for reliable, high‑performance messaging in production environments.