How to Build a Scalable Delayed Queue with Redis and Java

Background

During business operations, scenarios such as unpaid orders after 30 minutes, default comments after 48 hours, or unassigned delivery orders after a timeout require delayed processing. Simple polling works for small data volumes but becomes resource‑intensive at scale, leading to slow queries or timeouts, so a delayed queue is preferred.

2. Types of Delayed Queues

Delayed queues are message queues with built‑in delay capabilities. The following implementations are common:

Java java.util.concurrent.DelayQueue Pros: Built into JDK, easy to use, lightweight. Cons: Messages reside in JVM memory, no distributed support or persistence.

RocketMQ Delayed Queue Pros: Message persistence, distributed. Cons: Only supports predefined delay levels, not arbitrary precision.

RabbitMQ Delayed Queue (TTL + DLX) Pros: Message persistence, distributed. Cons: Messages with the same delay must share a queue.

When building a custom delayed‑queue service, consider message storage, real‑time retrieval of expired messages, and high availability.

3. Redis‑Based Implementation

1.0 Version

Features

* Message reliability, persistence, at‑least‑once consumption * Real‑time: slight time deviation due to task interval * Support for explicit message removal * High availability

Overall Structure

Components:

Messages Pool – KV store (Redis Hash) where each key is a message ID and the value is the full message.

Delayed Queue – 16 sorted sets (ZSET) holding message IDs; the score is the expiration timestamp. Multiple queues improve scan speed.

Timed Task – background job that scans each queue for expired messages.

Message Structure

Each delayed message must contain:

tags – MQ tags after expiration

keys – MQ keys after expiration

body – payload for consumer processing

delayTime – delay duration (or expectDate)

expectDate – desired send time

Process Flow

1. Pull job threads (one per queue) fetch expired messages. 2. Workers process fetched messages, improving real‑time handling. 3. Zookeeper coordinates queue re‑allocation; daemon threads watch ZK node changes.

2.0 Version

The 1.0 version relied on a 1‑minute scheduled task to trigger expiration checks. The 2.0 version replaces this with a Java Lock.await() / Signal mechanism, enabling real‑time, low‑latency delivery of expired messages.

Multi‑Node Deployment

Key components:

Pull job – dedicated thread per queue that pulls expired messages.

Worker – processes messages immediately after pull, avoiding bottlenecks.

Zookeeper coordination – handles dynamic queue reassignment when nodes are added or removed.

Main Flow

When the service starts, it registers with ZK, obtains assigned queues, and launches background threads. For each queue, a pull job checks for expired messages:

If found, the message is handed to a worker.

If not, the job examines the last ZSET member; if none, it awaits; otherwise it awaits based on the score offset.

Pull jobs track the last offset to ensure messages are not lost after a failure. When nodes change, the service recalculates queue assignments, cancels old pull jobs, and creates new ones.