Design and Implementation of Delayed Messages in Distributed Message Queues

Introduction

Delayed (scheduled) messages refer to messages produced in a distributed asynchronous messaging scenario that are intended to be consumed at a specified future time rather than immediately.

Such functionality is widely used in distributed systems and is typically implemented at the middleware layer, often as a built‑in feature of the MQ or as a shared infrastructure service.

This article explores common implementation schemes for delayed messages and discusses the pros and cons of each design.

Implementation Schemes

1. External‑storage‑based solutions

External storage means storage systems other than the MQ's built‑in storage.

The core idea is to separate the MQ from a dedicated delay‑message service/process. Messages are first persisted in an external store and delivered to the MQ when their delivery time arrives.

Advantages: simple implementation.

Disadvantages: B+Tree indexes are not suitable for heavy write workloads.

Database (e.g., MySQL)

Using a relational table to hold delayed messages.

CREATE TABLE `delay_msg` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `delivery_time` DATETIME NOT NULL COMMENT '投递时间',
  `payloads` blob COMMENT '消息内容',
  PRIMARY KEY (`id`),
  KEY `time_index` (`delivery_time`)
);

The table is scanned by a scheduled thread; the scan interval defines the minimum time granularity.

RocksDB

RocksDB provides an LSM‑tree storage that is more write‑friendly. DDMQ’s Chronos module uses this approach.

Advantages: high write throughput suitable for massive insertions.

Disadvantages: heavier implementation, requiring custom data‑replication logic for durability.

Redis

A mature solution based on Redis Hashes for the message pool and ZSETs for delayed queues.

Workers periodically scan the ZSETs for expired messages.

Pros: O(log n) insertion, in‑memory speed, and good concurrency support.

Cons: needs distributed locking to avoid duplicate processing when multiple nodes share queues.

2. Open‑source MQ implementations

RocketMQ

Supports 18 fixed delay levels (e.g., 1 s, 5 s, …, 2 h). Delayed messages are stored in a special topic SCHEDULE_TOPIC_XXXX and dispatched to the real topic when the level expires.

Advantages: low overhead, preserves order per level, and each level has its own timer.

Disadvantages: inflexible level configuration, large commit log due to delayed messages.

Pulsar

Supports arbitrary delay times by keeping an off‑heap priority queue per subscription group.

Advantages: fine‑grained delays.

Disadvantages: high memory consumption (one queue per subscription group), costly recovery after broker failure, and storage impact for long‑delay messages.

QMQ

Provides arbitrary‑time delayed messages using a two‑level hierarchical time wheel, schedule logs on disk, and an in‑memory wheel with 500 ms slots.

Key advantages: O(1) insertion/deletion, support for multi‑year delays, low memory footprint due to delayed loading, and separate storage for delayed messages that does not affect normal message reclamation.

Conclusion

The article summarizes the most common delayed‑message designs in the industry, compares their strengths and weaknesses, and hopes to inspire readers when choosing or building a delayed‑message solution.