Research on Delayed Queue Technologies and Implementation Approaches

Delayed queues are message queues with built‑in latency, useful for scenarios that require processing after a certain wait period. The article first outlines the project background and the need for such a mechanism.

Kafka solution: By leveraging KafkaConsumer.pause() and resume() , consumers can temporarily halt processing when the scheduled time has not arrived, moving offsets back to the last committed position. This approach incurs high internal complexity, limited flexibility in setting delay durations, and requires additional health‑check logic.

RocketMQ solution: RocketMQ provides 18 predefined delay levels; messages are sent to a special topic (e.g., delay‑minutes‑1 ) and a timer periodically scans these topics, moving expired messages to the target topic. Advantages include simple level‑based sorting, but the single‑threaded timer can become a bottleneck under heavy load.

Redis (Redisson) solution: Redisson implements delayed queues using three core data structures: a delay queue (list), a blocking queue for ready messages, and a zset (timeoutSet) that stores expiration timestamps. A timer thread subscribes to a Pub/Sub channel, polls the zset for expired entries, and moves them to the blocking queue. Drawbacks are potential duplicate timer execution across clients and the need for a cluster mode with distributed locks for high availability.

Youzan's delayed‑queue design: The system defines a job (unique ID, topic, delayTime, ttr, message), a job pool (map of original job data), a delay bucket (Redis zset storing job IDs ordered by execution time), a timer that scans buckets and moves due jobs to a ready queue , and finally a Kafka topic that serves as the common exit point for consumers. The flow is: business services push jobs to an entrance topic → timer moves due jobs to Kafka → consumers read from the shared Kafka topic.

Netty HashedWheelTimer: This time‑wheel implementation uses an array of buckets ( ticksPerWheel ) where each bucket holds timer tasks. The wheel advances at a fixed tickDuration , executing tasks whose deadline has passed. Core loop example:

long deadline = tickDuration * (tick + 1);
for (;;) {
    final long currentTime = System.nanoTime() - startTime;
    long sleepTimeMs = (deadline - currentTime + 999999) / 1000000;
    if (sleepTimeMs <= 0) {
        if (currentTime == Long.MIN_VALUE) {
            return -Long.MAX_VALUE;
        } else {
            return currentTime;
        }
    }
    if (PlatformDependent.isWindows()) {
        sleepTimeMs = sleepTimeMs / 10 * 10;
    }
    try {
        Thread.sleep(sleepTimeMs);
    } catch (InterruptedException ignored) {
        if (WORKER_STATE_UPDATER.get(HashedWheelTimer.this) == WORKER_STATE_SHUTDOWN) {
            return Long.MIN_VALUE;
        }
    }
}

Kafka time‑wheel: Builds on the standard time‑wheel by storing each bucket in a DelayQueue (which internally uses a PriorityQueue ordered by bucket expiration). The timer polls the DelayQueue , advances the wheel, and flushes expired buckets. Core logic:

private[this] val reinsert = (timerTaskEntry: TimerTaskEntry) => addTimerTaskEntry(timerTaskEntry)

def advanceClock(timeoutMs: Long): Boolean = {
    var bucket = delayQueue.poll(timeoutMs, TimeUnit.MILLISECONDS)
    if (bucket != null) {
        writeLock.lock()
        try {
            while (bucket != null) {
                timingWheel.advanceClock(bucket.getExpiration())
                bucket.flush(reinsert)
                bucket = delayQueue.poll()
            }
        } finally {
            writeLock.unlock()
        }
        true
    } else {
        false
    }
}

The article concludes that designing a delayed‑queue system hinges on two core problems: (1) how to sort or index delayed tasks efficiently, and (2) how to detect and trigger execution when the scheduled time arrives. Various solutions address these differently—bucket‑sorting in RocketMQ, array‑based sorting in HashedWheelTimer, and zset‑based ordering in Redis—allowing engineers to choose or combine approaches based on their specific requirements.

Overall, the survey provides a comprehensive comparison of delayed‑queue technologies, highlights their trade‑offs, and presents practical implementation patterns for building reliable, scalable delayed messaging in backend systems.