Common Implementation Schemes for Delayed Messages in Distributed Systems

Delayed messages (scheduled messages) refer to messages sent by producers that should be consumed at a specified future time rather than immediately, commonly used in distributed asynchronous messaging scenarios.

They are typically implemented at the middleware layer, often within MQ or as a separate service.

Implementation Schemes

External Storage Based

In this approach, a separate delayed‑message module stores messages in an external storage (e.g., MySQL, RocksDB, Redis) and a timer thread scans for expired messages to deliver them to the MQ.

Database (MySQL)

Uses a delay_msg table with columns id , delivery_time , payloads . A scheduled thread periodically scans the table and delivers due messages.

CREATE TABLE `delay_msg` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `delivery_time` DATETIME NOT NULL COMMENT '投递时间',
  `payloads` blob COMMENT '消息内容',
  PRIMARY KEY (`id`),
  KEY `time_index` (`delivery_time`)
);

Advantages: simple implementation. Drawbacks: B‑Tree index not suitable for high‑write message workloads.

RocksDB

Uses LSM‑Tree storage, suitable for heavy writes. Example: DDMQ’s Chronos module stores delayed messages in RocksDB and periodically moves them to RocketMQ.

Advantages: high write performance. Drawbacks: heavier solution requiring custom replication and data‑sync logic.

Redis

Stores messages in a hash (Messages Pool) and uses multiple sorted‑set queues (Delayed Queue) where the score is the expiration timestamp. Workers scan the queues and deliver due messages.

Advantages: ZSET provides O(log n) insertion and fast in‑memory access. Drawbacks: potential duplicate processing without distributed locks.

Timer‑Thread Defects and Improvements

Traditional timer threads may waste resources or cause inaccurate delays. Using a wait‑notify mechanism similar to JDK Timer can reduce CPU usage by sleeping until the next message’s delivery time.

Open‑Source MQ Implementations

RocketMQ

Supports 18 fixed delay levels (e.g., 1 s, 5 s, …, 2 h). Delayed messages are stored in a special topic SCHEDULE_TOPIC_XXXX with a queue per level. Broker periodically moves messages to the target topic.

Advantages: low overhead, ordered delivery per level. Drawbacks: inflexible level configuration and increased CommitLog size.

Pulsar

Supports arbitrary delay times by storing messages in the target topic and maintaining an off‑heap priority queue for indexes. On consumption, the broker checks the queue for due messages.

Drawbacks: high memory usage for indexes, costly failover reconstruction, and storage impact for long‑delay spans.

QMQ

Provides arbitrary‑time delayed messages using a two‑level hierarchical time wheel: a disk‑based hour‑level wheel and an in‑memory 500 ms wheel. This design offers O(1) operations, supports large time spans, and isolates delayed messages from normal traffic.

Key benefits: efficient time‑wheel algorithm, multi‑level wheels for long delays, delayed loading to keep memory usage low, and separate storage for delayed messages.

The article summarizes common delayed‑message solutions, compares their pros and cons, and offers practical guidance for selecting an appropriate scheme.