Implementation Strategies for Delayed (Scheduled) Messages in Distributed Systems

Introduction

Delayed (scheduled) messages are used in distributed asynchronous messaging where a producer wants a message to be consumed at a specific future time rather than immediately.

They are typically implemented at the middleware layer, often within the MQ or as a separate service.

Implementation Approaches

1. External Storage Based Solutions

Database (e.g., MySQL)

Store delayed messages in a relational table and use a scanning thread to deliver due messages.

CREATE TABLE `delay_msg` ( `id` bigint unsigned NOT NULL AUTO_INCREMENT, `delivery_time` DATETIME NOT NULL COMMENT '投递时间', `payloads` blob COMMENT '消息内容', PRIMARY KEY (`id`), KEY `time_index` (`delivery_time`) )

Advantages: simple implementation. Drawbacks: B+Tree index not suited for heavy writes.

RocksDB

Uses LSM‑tree storage suitable for high‑write workloads; example DDMQ Chronos stores delayed messages in RocksDB and forwards them to RocketMQ when due.

Advantages: high write performance. Drawbacks: heavier solution requiring custom replication and fault‑tolerance logic.

Redis

Implements a pool of messages (hash) and multiple ZSET queues for delayed IDs; workers scan ZSETs and deliver messages.

Advantages: O(log n) insertion, in‑memory speed. Drawbacks: potential duplicate processing and need for distributed locks.

2. Built‑in MQ Implementations

RocketMQ

Supports 18 fixed delay levels (e.g., 1 s, 5 s, …, 2 h). Messages are stored in a special topic SCHEDULE_TOPIC_XXXX and dispatched to the real topic when the level expires.

Advantages: low overhead, ordered delivery per level. Drawbacks: inflexible level configuration and larger commit log.

Pulsar

Allows arbitrary delay times by keeping an off‑heap priority queue per subscription group; messages are moved to the target topic when due.

Drawbacks: high memory usage, costly recovery after failures, and storage impact due to retained messages.

QMQ

Uses a two‑level hierarchical time wheel (disk‑based hourly slots and in‑memory 500 ms slots) to achieve O(1) insert/delete and support multi‑year delays.

Advantages: efficient O(1) operations, scalable time span, memory‑friendly delayed loading, and separate storage for delayed messages.

Conclusion

The article summarizes prevalent delayed‑message designs, highlighting their trade‑offs to help practitioners choose an appropriate solution.