Evolving Message Expiration for 10 Million QPS: From No TTL to Smart Policies

In a large‑scale e‑commerce promotion, the monitoring team observed that after the traffic peak the consumer throughput remained high while the backlog of a critical order topic barely decreased. The backlog consisted of "zombie" messages that were hours or days old, causing order cancellations to be processed incorrectly and leading to user‑visible errors.

Midnight anomaly

The incident revealed that a message queue is not an archive; without a freshness guarantee, stale messages become a corrosive factor in the processing pipeline.

The "never expire" era

Early implementations left messages without any expiration. Operators extended retention from 1 day to 30 days to satisfy observability and replay requirements. This worked while the system was small, but during a downstream service outage the queue accumulated tens of millions of messages, causing order creation to be delayed and cancellations to be processed out of order.

Step 1: Uniform TTL

The first mitigation added a single expire_at field to every message and let brokers or consumers drop messages whose timestamp was past the configured TTL (e.g., 1 hour or 6 hours). This created a clear boundary: after a failure, only recent messages were consumed, and the rest were sent to a dead‑letter queue for later compensation.

However, a single TTL acted like an un‑graded scissors, discarding both truly obsolete messages and partially processed business data. Extending the TTL to avoid loss re‑introduced the original accumulation problem.

Step 2: Per‑topic TTL

TTL was split by topic: order‑related topics received a 24‑hour TTL, monitoring topics 5 minutes, and alarm topics 10 seconds. In Kafka this is configured via retention.ms, in RocketMQ via messageDelayLevel and MessageExpired, and in RabbitMQ via x-message-ttl on the queue.

While this solved many cross‑topic interference issues, it still could not express the differing decay curves of individual messages within the same topic, leading to either value loss or resource waste.

Step 3: Smart expiration

Smart expiration moves the decision from a static TTL to a dynamic, business‑context check. It consists of three actions:

Explicit validity attributes : producers attach "effective time" and "invalid‑condition" metadata (e.g., dependent order status, inventory state, user online flag). Consumers perform a lightweight check and discard messages whose pre‑conditions have already failed.

Consumption‑speed‑aware policies : the broker monitors per‑shard latency; if a shard lags beyond a threshold, messages on that shard are marked with a high‑expiration‑risk flag and processed via a degraded path (skip non‑critical steps, merge adjacent messages, or route to dead‑letter).

Auditable expiration decisions : every expiration decision is logged with reason, topic, and consumer group, allowing replay and, if necessary, recovery of mistakenly expired messages from the dead‑letter queue.

TTL indexing and cleanup at 10 M QPS

Two implementation styles exist:

Lazy expiration (e.g., Redis): TTL is checked only when a message is read, keeping write and cleanup overhead low but allowing expired data to occupy storage.

Active expiration (e.g., Kafka LogCleaner, RocketMQ scanning thread): a background thread periodically scans and removes expired messages, which is timely but incurs heavy I/O at billions of messages per day.

In practice, both must be combined with tiered granularity: hot topics use lazy expiration, warm topics use active expiration with reduced scan frequency, and cold data is archived and cleaned offline.

Maintaining per‑message expiration indexes also incurs CPU and memory cost; a common compromise is to bucket timestamps into 1‑minute or 10‑second buckets, providing minute‑level precision without exploding index size.

Separating dead‑letter, expiration, and retry

Dead‑letter queues hold messages that have permanently failed after retries; expiration queues contain messages that have lost business value over time. Mixing them leads to dead‑letter overload and ambiguous compensation logic. The recommended approach is three independent paths, each with its own queue, monitoring, and handling strategy.

Smart expiration implementation checklist

Propagate three timestamps (business event time, enqueue time, expiration time) in every message.

Consumers perform a second local validation after broker TTL check.

Make expiration policies configurable and gradable per topic, sub‑type, or business tag, with safe rollout via canary deployment.

Audit all expired messages, aggregating by topic, reason, and consumer group for early‑warning signals.

Define downgrade paths (drop, delayed drop, compensation) in advance with business owners.

Tier TTL index and cleanup: lazy for hot data, time‑bucket scans for warm data, offline archiving for cold data.

Keep dead‑letter, expiration, and retry pipelines completely independent.

When "should not expire" messages still need a lifecycle

Even seemingly permanent data such as financial ledgers or compliance records require a clear retention or archival policy, typically at day‑ or month‑level granularity, to avoid unbounded storage growth.

In summary, message expiration evolves from a naive "never expire" stance to uniform TTL, then tiered TTL, and finally to intelligent, context‑aware expiration. At the 10 M QPS scale, expiration becomes a full subsystem interacting with monitoring, configuration, and audit platforms, essential for maintaining system order and business boundaries.