Design and Implementation Strategies for Delayed Messages in Distributed Message Queues

Implementation Schemes

External Storage Based Schemes

External storage refers to any storage system introduced besides the MQ's native storage. The basic pattern separates the MQ from a dedicated delay‑module service/process that stores messages in another medium until they expire, then forwards them to the MQ.

Different storage systems are used in the following variants.

Based on Relational Databases (e.g., MySQL)

Using a delay‑message table in a relational DB.

CREATE TABLE `delay_msg` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `delivery_time` DATETIME NOT NULL COMMENT '投递时间',
  `payloads` blob COMMENT '消息内容',
  PRIMARY KEY (`id`),
  KEY `time_index` (`delivery_time`)
)

A scheduled thread scans for expired rows at a configurable interval, which defines the minimum delay granularity.

Pros: Simple to implement.

Cons: B‑Tree indexes are not optimal for high‑write message workloads.

Based on RocksDB

RocksDB uses an LSM‑Tree, which is write‑friendly. DDMQ’s Chronos module adopts this approach: delayed messages are first stored in a dedicated RocketMQ topic, then consumed by Chronos and persisted in RocksDB, where a timer scans and re‑delivers them.

Pros: LSM‑Tree handles massive writes efficiently.

Cons: The solution is heavyweight; you must implement your own replication and disaster‑recovery logic for RocksDB.

Based on Redis

A more complete Redis design uses a message pool (Hash) for storing payloads and 16 ordered ZSET queues (Delayed Queues) for timestamps. Workers periodically scan the ZSETs and deliver expired messages.

Pros: Redis ZSET naturally fits delayed queues; in‑memory operations are fast.

Cons: Managing multiple queues across nodes can cause uneven load distribution and duplicate processing without additional coordination (e.g., distributed locks).

Drawbacks of Simple Timer Threads and Improvements

All the above schemes rely on periodic scanning threads, which waste resources at low load and may cause inaccurate delays at high load. A more efficient approach uses a wait‑notify mechanism similar to JDK Timer: the thread waits until the next message’s delivery time, wakes early if a sooner message arrives, and repeats.

Open‑Source MQ Implementations

RocketMQ

RocketMQ supports delayed messages via 18 configurable levels (e.g., 1s, 5s, …, 2h). Messages are stored in a special topic SCHEDULE_TOPIC_XXXX with a queue per level, preserving order within each level. A broker periodically moves messages from the schedule topic to the target topic.

Pros: Fixed levels keep overhead low; same‑level messages stay in one queue, guaranteeing order and timing accuracy.

Cons: Changing level configuration is costly; delayed messages increase CommitLog size.

Pulsar

Pulsar allows arbitrary delay times. Delayed messages are indexed in an off‑heap priority queue per subscription group; the earliest message sits at the head. On consumption, the broker checks the queue and delivers messages whose delay has elapsed.

Cons: High memory usage (one queue per subscription group), costly failover reconstruction, and storage overhead because delayed messages keep their data for the whole delay span.

QMQ

QMQ provides arbitrary‑time delayed/scheduled messages using a two‑level hierarchical time wheel plus delayed loading and separate disk storage.

The first level is a disk‑based hour wheel (one log file per hour, up to two years). The second level is an in‑memory wheel with 500 ms slots that loads the nearest hour’s index on demand.

Pros: O(1) insert/delete via time‑wheel, supports very long delays, memory‑friendly due to delayed loading, and stores delayed messages separately from normal traffic.

In summary, the article surveys common delayed‑message designs, compares their strengths and weaknesses, and offers guidance for selecting an appropriate solution in distributed systems.