How to Build a Redis-Based Delayed Queue with Lua Scripts

Introduction

A delayed queue stores messages that are consumed only after a specified delay, unlike a normal queue where messages are processed immediately.

Difference Between Delayed Tasks and Scheduled Tasks

Scheduled tasks run at fixed intervals with a known trigger time.

Delayed tasks are triggered by an event and fire after a configurable period; they have no fixed start time.

The event is generated but not delivered to the consumer until the delay expires.

Typical Business Scenarios

Order timeout: cancel an order if payment is not completed within a set period (e.g., 15 minutes).

Periodic verification of refund status.

Send reminder SMS/email to users who have been registered for a week but remain inactive.

Double‑check transaction information to avoid loss caused by system or user anomalies.

Retry notifications up to 10 times with an interval of n*2+1/min, marking the task as abnormal after the limit.

Use Cases

Delayed Consumption : verify payment status after order creation; if unpaid, close the order. Verify user activity a week after registration and send a reminder if needed.

Delayed Retry : when a consumer fails to process a message, re‑queue it for later retry instead of writing a separate polling program.

Cache Queue Design

Scenario Design

In a production system each payment request must push an attachment after a 30‑minute delay. The simplified example records an OrderMessage for each order and processes it 5–15 seconds later.

Implementation of the Delayed Queue

The solution uses Redis Sorted Set to store execution timestamps, a Hash to store payloads, and a short‑polling Crontab job to trigger consumption.

Specific Steps

Add the order ID as a member and the execution timestamp as the score to a Sorted Set.

Store the order ID as the field and the JSON payload as the value in a Hash.

Execute steps 1 and 2 atomically with a Lua script.

In an asynchronous worker call ZREVRANGEBYSCORE on the Sorted Set, pop a batch of IDs, fetch the corresponding payloads from the Hash, process them, and delete the entries.

Options for Step 4

Option 1 – Pop the data and delete it in the same Lua script using ZREVRANGEBYSCORE, ZREM and HDEL. This keeps the operation atomic but makes the script more complex; if processing fails, compensation from a database may be required.

Option 2 – Pop the data first, then delete the Sorted Set and Hash entries after successful processing. This requires careful concurrency control to avoid duplicate execution.

The implementation chooses Option 1 : after popping the order ID, the payload is fetched from the Hash and both structures are deleted immediately.

Redis Commands

Sorted Set Commands

ZADD – Add one or more members with scores to a sorted set.

ZADD KEY SCORE1 VALUE1 ... SCORE_N VALUE_N

ZREVRANGEBYSCORE – Return members in a score range in descending order.

ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]

max

– maximum score. min – minimum score. WITHSCORES – optional, also return scores. LIMIT – works like MySQL LIMIT offset,count.

ZREM – Remove one or more members from a sorted set.

ZREM key member [member ...]

Hash Commands

HMSET – Set multiple field‑value pairs in a hash.

HMSET KEY_NAME FIELD1 VALUE1 ... FIELD_N VALUE_N

HDEL – Delete one or more fields from a hash.

HDEL KEY_NAME FIELD1 ... FIELD_N

Lua Scripting Basics

Load a script and obtain its SHA‑1: SCRIPT LOAD script.

Execute a loaded script: EVALSHA sha1 numkeys key [key ...] arg [arg ...]. unpack converts a Lua table to var‑args; it must be the last argument in a non‑variable‑definition call.

Lua Scripts

Enqueue ( enqueue.lua )

local zset_key = KEYS[1]
local hash_key = KEYS[2]
local zset_value = ARGV[1]
local zset_score = ARGV[2]
local hash_field = ARGV[3]
local hash_value = ARGV[4]
redis.call('ZADD', zset_key, zset_score, zset_value)
redis.call('HSET', hash_key, hash_field, hash_value)
return nil

Dequeue ( dequeue.lua )

local zset_key = KEYS[1]
local hash_key = KEYS[2]
local min_score = ARGV[1]
local max_score = ARGV[2]
local offset = ARGV[3]
local limit = ARGV[4]
local status, type = next(redis.call('TYPE', zset_key))
if status ~= nil and status == 'ok' then
    if type == 'zset' then
        local list = redis.call('ZREVRANGEBYSCORE', zset_key, max_score, min_score, 'LIMIT', offset, limit)
        if list ~= nil and #list > 0 then
            redis.call('ZREM', zset_key, unpack(list))
            local result = redis.call('HMGET', hash_key, unpack(list))
            redis.call('HDEL', hash_key, unpack(list))
            return result
        end
    end
end
return nil

If the smallest score is less than or equal to the current timestamp, the task is taken for execution; otherwise the consumer sleeps and retries later.

Performance note: ZREVRANGEBYSCORE has O(N) complexity, which can become a bottleneck when the sorted set grows large.

Core PHP Implementation ( RedisDelayQueue.php )

<?php
declare(strict_types=1);
namespace redis;
class RedisDelayQueue {
    const DELAY_QUEUE_PRODUCER_SCRIPT_SHA = 'DELAY:QUEUE:PRODUCER:SCRIPT:SHA';
    const DELAY_QUEUE_CONSUMER_SCRIPT_SHA = 'DELAY:QUEUE:CONSUMER:SCRIPT:SHA';
    const DELAY_QUEUE_ORDER_CLOSE = 'DELAY:QUEUE:ORDER:CLOSE';
    const DELAY_QUEUE_ORDER_CLOSE_HASH = 'DELAY:QUEUE:ORDER:CLOSE:HASH';

    private static function _redis() {
        $redis = \redis\BaseRedis::server();
        $redis->select(3);
        return $redis;
    }

    public static function producer(string $keys1, string $keys2, string $member, int $score, array $message) {
        $redis = self::_redis();
        $scriptSha = $redis->get(self::DELAY_QUEUE_PRODUCER_SCRIPT_SHA);
        if (!$scriptSha) {
            $script = "redis.call('ZADD', KEYS[1], ARGV[1], ARGV[2])
redis.call('HSET', KEYS[2], ARGV[2], ARGV[3])
return 1";
            $scriptSha = $redis->script('load', $script);
            $redis->set(self::DELAY_QUEUE_PRODUCER_SCRIPT_SHA, $scriptSha);
        }
        $hashValue = json_encode($message, JSON_UNESCAPED_UNICODE);
        return $redis->evalSha($scriptSha, [$keys1, $keys2, $score, $member, $hashValue], 2);
    }

    public static function consumer(string $keys1, string $keys2, int $maxScore) {
        $redis = self::_redis();
        $scriptSha = $redis->get(self::DELAY_QUEUE_CONSUMER_SCRIPT_SHA);
        if (!$scriptSha) {
            $script = "local status, type = next(redis.call('TYPE', KEYS[1]))
if status ~= nil and status == 'ok' then
  if type == 'zset' then
    local list = redis.call('ZREVRANGEBYSCORE', KEYS[1], ARGV[1], ARGV[2], 'LIMIT', ARGV[3], ARGV[4])
    if list ~= nil and #list > 0 then
      redis.call('ZREM', KEYS[1], unpack(list))
      local result = redis.call('HMGET', KEYS[2], unpack(list))
      redis.call('HDEL', KEYS[2], unpack(list))
      return result
    end
  end
end";
            $scriptSha = $redis->script('load', $script);
            $redis->set(self::DELAY_QUEUE_CONSUMER_SCRIPT_SHA, $scriptSha);
        }
        return $redis->evalSha($scriptSha, [$keys1, $keys2, $maxScore, 0, 0, 10], 2);
    }
}

Command‑Line Usage

Producer Message

private function delayQueueOrderClose() {
    $orderId = time();
    $keys1 = RedisDelayQueue::DELAY_QUEUE_ORDER_CLOSE;
    $keys2 = RedisDelayQueue::DELAY_QUEUE_ORDER_CLOSE_HASH;
    $score = time() + 60; // delay 60 seconds
    $message = [
        'event' => RedisDelayQueue::EVENT_ORDER_CLOSE,
        'order_id' => $orderId,
        'create_time' => time()
    ];
    $res = RedisDelayQueue::producer($keys1, $keys2, (string)$orderId, $score, $message);
    var_dump($res);
}

Consumer Message (Polling)

private function delayQueueOrderConsumer() {
    $keys1 = RedisDelayQueue::DELAY_QUEUE_ORDER_CLOSE;
    $keys2 = RedisDelayQueue::DELAY_QUEUE_ORDER_CLOSE_HASH;
    $maxScore = time();
    $queueList = RedisDelayQueue::consumer($keys1, $keys2, $maxScore);
    if (false === $queueList) {
        echo " [x] Message List is Empty, Try Again 
";
        return;
    }
    var_dump($queueList);
}

Consumer Message (Blocking Loop)

private function delayQueueOrderConsumerWhile() {
    $keys1 = RedisDelayQueue::DELAY_QUEUE_ORDER_CLOSE;
    $keys2 = RedisDelayQueue::DELAY_QUEUE_ORDER_CLOSE_HASH;
    while (true) {
        $maxScore = time();
        $queueList = RedisDelayQueue::consumer($keys1, $keys2, $maxScore);
        if (false === $queueList) {
            echo " [x] Message List is Empty, Try Again 
";
            sleep(1);
            continue;
        }
        foreach ($queueList as $queue) {
            $messageArray = json_decode($queue, true);
            // process business logic here
        }
    }
}

Data Deletion and At‑Least‑Once Guarantee

Option 1 deletes the order data from the Sorted Set and Hash inside the same Lua script using ZREVRANGEBYSCORE , ZREM and HDEL . This keeps the operation atomic but requires compensation if processing fails.

To ensure each delayed‑queue message is consumed at least once, the workflow can be extended:

After a message is read from the Sorted Set, push it into a Redis Stream and add it to a consumer group.

Consume the stream via the consumer group, execute business logic, and acknowledge with ACK.

If a message is read but not processed, it appears in the XPENDING list for a second‑chance consumption before being acknowledged.