How to Build a Reliable Distributed Delay Queue in Java: Strategies, Pros & Cons

Application Scenarios

Many functions in the system require delayed processing, such as payment timeout cancellation, queue timeout, delayed SMS/WeChat notifications, token refresh, membership card expiration, etc. Using delayed processing saves resources by avoiding database polling.

Most functions currently rely on scheduled tasks (Quartz or XXL‑Job) that poll every second, causing database pressure and up to one‑second error. Longer intervals (e.g., every 30 minutes) can introduce delays of up to 29 minutes.

Research of Processing Methods

1. DelayQueue

Implementation: Java provides a blocking delay queue that orders tasks by delay time using a priority queue and blocks with a condition until the delay expires.

When a new task arrives, if it becomes the earliest task, the queue is awakened to ensure timely execution.

Problems:

Single‑node operation; no retry after a crash.

No execution records or backups.

No retry mechanism.

Tasks are cleared on system restart.

Cannot be sharded for consumption.

Advantages: Simple implementation, blocks when idle, saves resources, and provides accurate execution timing.

2. MQ‑based Delay Queue

Implementation: Relies on message‑queue features that allow setting a delayed consumption time (e.g., RabbitMQ, JMQ). RabbitMQ can set message expiration and place it in a dead‑letter queue for later consumption.

Problems: Inflexible time settings; each queue has a fixed TTL, requiring a new queue for each delay configuration.

Advantages: MQ provides monitoring, consumption records, retries, and multi‑node consumption, making it resilient to crashes.

3. Scheduled Tasks

Polls the database for tasks that meet the condition.

Drawbacks:

Creates database pressure.

Introduces latency.

Large scans consume excessive system resources.

Cannot be sharded for consumption.

Advantages:

Failed consumption can be retried.

Stable consumption capability.

4. Redis ZSET

Tasks are stored in a Redis sorted set (ZSET) with the execution timestamp as the score. The program continuously fetches tasks whose score is less than the current time.

Advantages:

Redis queries are faster than database scans; ZSET uses a skip‑list for efficient range queries.

Can sort by timestamp and retrieve only due tasks.

Resilient to node restarts.

Supports distributed consumption.

Drawbacks:

Limited by Redis performance; high concurrency may hit 100 k QPS limits.

Multiple commands are not atomic without Lua scripts, which require all data on a single Redis shard.

5. Time Wheel

Implements delayed execution using a time wheel (e.g., Netty’s HashedWheelTimer). Suitable for single‑node JVM but not for distributed services; tasks are lost on crash.

Drawbacks: Not suitable for distributed environments; tasks disappear after a node failure.

Implementation Goals

Provide a component compatible with existing asynchronous event infrastructure that offers reliable, retry‑capable, recorded, monitorable, and high‑performance delayed processing.

Message transmission reliability: at least once consumption.

Rich client support for multiple languages.

High availability with multi‑instance deployment.

Acceptable real‑time deviation.

Support for message deletion.

Support for consumption queries and manual retries.

Monitoring of asynchronous event execution.

Architecture Design

Delay Component Implementation

1. Implementation Principle

Use JimDB’s ZSET to store task IDs with execution timestamps as scores. When sending a delayed task, generate a unique ID (timestamp + IP + queue name + sequence), encrypt the payload, and add it to the ZSET.

A mover thread transfers tasks whose execution time has arrived from the ZSET to a publish queue for consumers.

Monitoring integrates with UMP; consumption records are backed up in Redis and persisted to a database.

2. Message Structure

Topic: job type (queue name).

Id: unique job identifier.

Delay: delay time in seconds (converted to absolute time on the server).

Body: JSON payload for business processing.

traceId: trace identifier of the sending thread.

3. Data Flow and Diagram

The system uses a Redis‑Disruptor model for publishing and consuming messages. Consumers employ the existing asynchronous event Disruptor lock‑free queue, allowing different applications and queues to operate without locking.

Supports publish‑only mode, acting like a message queue.

Supports sharding to mitigate large‑key issues in Redis.

Performance can be scaled via DUCC configuration (enable/disable consumption).

Configurable timeout to prevent excessively long consumer execution.

Bottlenecks: Consumption speed may become a bottleneck if production outpaces consumption, filling the ring buffer and causing producer back‑pressure. Monitoring Redis queue length and adding consumers can alleviate this.

Demo Example

Configuration

<dependency>
  <groupId>com.jd.car</groupId>
  <artifactId>senna-event</artifactId>
  <version>1.0-SNAPSHOT</version>
</dependency>

jd:
  senna:
    event:
      enable: true
      queue:
        retryEventQueue:
          bucketNum: 1
          handleBean: retryHandle

Consumer Code

package com.jd.car.senna.admin.event;

import com.jd.car.senna.event.EventHandler;
import com.jd.car.senna.event.annotation.SennaEvent;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;

/**
 * @author zhangluyao
 */
@Slf4j
@Component("retryHandle")
public class RetryQueueEvent extends EventHandler {
    @Override
    protected void onHandle(String key, String eventType) {
        log.info("Handler starts processing:{}", key);
    }

    @Override
    protected void onDelayHandle(String key, String eventType) {
        log.info("DelayHandler starts processing:{}", key);
    }
}

Annotation‑Based Consumer

package com.jd.car.senna.admin.event;

import com.jd.car.senna.event.EventHandler;
import com.jd.car.senna.event.annotation.SennaEvent;
import lombok.extern.slf4j.Slf4j;

/**
 * @author zhangluyao
 */
@Slf4j
@SennaEvent(queueName = "testQueue", bucketNum = 5, delayBucketNum = 5, delayEnable = true)
public class TestQueueEvent extends EventHandler {
    @Override
    protected void onHandle(String key, String eventType) {
        log.info("Handler starts processing:{}", key);
    }

    @Override
    protected void onDelayHandle(String key, String eventType) {
        log.info("DelayHandler starts processing:{}", key);
    }
}

Sending Code

package com.jd.car.senna.admin.controller;

import com.jd.car.senna.event.queue.IEventQueue;
import lombok.extern.slf4j.Slf4j;
import org.springframework.context.annotation.Lazy;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;
import javax.annotation.Resource;
import java.util.Date;

/**
 * Demo controller
 */
@RestController
@Slf4j
public class DemoController {
    @Lazy
    @Resource(name = "testQueue")
    private IEventQueue eventQueue;

    @ResponseBody
    @GetMapping("/api/v1/demo")
    public String demo() {
        log.info("Send immediate message");
        eventQueue.push("no delay 5000 milliseconds message 3");
        return "ok";
    }

    @ResponseBody
    @GetMapping("/api/v1/demo1")
    public String demo1() {
        log.info("Send delayed 5‑second message");
        eventQueue.push("delay 5000 milliseconds message,name", 1000L * 5);
        return "ok";
    }

    @ResponseBody
    @GetMapping("/api/v1/demo2")
    public String demo2() {
        log.info("Send message delayed until 2022‑04‑02 00:00:00");
        eventQueue.push("delay message,name to 2022-04-02 00:00:00", new Date(1648828800000L));
        return "ok";
    }
}

Current Applications

Automatic cancellation of store queue after 24 hours.

Meituan token refresh requests.

Warranty card generation after 24 hours.

Settlement order delayed generation.

Delayed SMS sending.