Implementing Fixed-Length Queues and Batch Consumption in Redis with Lua

Business Background

The system must collect and report player‑behavior data per game, supporting up to 128 behaviors per game in batch form and allowing partial data loss. Reported data must be limited to the last three minutes, as illustrated in the architecture diagram.

Technical Options

Two approaches were considered. The first used traditional MQs such as RocketMQ or Kafka, but they could not satisfy the required per‑game grouping, batch processing, and timeliness constraints.

The second approach leveraged Redis lists to implement a fixed‑length queue, using LLEN, LPOP, and RPUSH for length control, LRANGE for batch reads, and multithreaded consumers to meet timeliness. This solution was chosen.

Solution Overview

Use a Redis List per game to store behavior messages, enforcing a maximum length.

Maintain a master List that records all games that currently have messages.

Use a Set to deduplicate game entries in the master list.

Architecture diagrams show the overall design.

Production Process

Push a behavior record into the game‑specific list.

If the game is not already present in the master set, add it to the master list.

If the game already exists, skip the addition.

Consumption Process

Iterate over the master game list to select a game.

For the selected game, retrieve a batch of messages from its list and process them.

Technical Principles

Redis native commands for List and Set are combined with Lua scripts to guarantee atomic execution.

Lua Script Characteristics

All commands inside a Lua script run atomically, providing thread‑safety.

The script implements the fixed‑length queue and batch consumption using list commands.

Lua scripts operate on a single key; multi‑key operations are not supported.

List Commands

LLEN key                // O(1) – get list length</code><code>LPOP key [count]       // O(N) – remove N elements from the left</code><code>RPUSH key element ...  // O(N) – push elements to the right</code><code>LRANGE key start end   // O(S+N) – return N elements starting at offset S</code><code>LTRIM key start end    // O(N) – keep only the specified range

By checking LLEN against the configured maximum, the script either pushes a new element or pops the oldest one before pushing, then sets an expiration.

Set Commands

SADD key member ...   // O(1) – add members</code><code>SISMEMBER key member // O(1) – test membership

The set guarantees uniqueness of game identifiers.

Application Details

Message Production

-- Lua script to enforce fixed‑length queue and expiration
local retVal = 0
local key = KEYS[1]
local maxLen = tonumber(ARGV[1])
local value = ARGV[2]
local ttl = tonumber(ARGV[3])
if redis.call('llen', key) < maxLen then
  redis.call('rpush', key, value)
else
  redis.call('lpop', key)
  redis.call('rpush', key, value)
  retVal = 1
end
redis.call('expire', key, ttl)
return retVal

The script atomically maintains the queue length and sets a TTL.

Message Consumption

-- Lua script for batch consumption
local key = KEYS[1]
local batch = tonumber(ARGV[1])
local data = redis.call('lrange', key, 0, batch)
redis.call('ltrim', key, batch+1, -1)
return data

The script returns a fixed number of messages and trims the list, both atomically.

Operational Notes

In Redis Cluster mode, Lua scripts should operate on a single key to avoid redirection errors.

Different Redis versions handle null returns differently; consult the official documentation.

Hot games receive dedicated consumer threads to prioritize processing.

Online Performance

Production and consumption sustain roughly 10,000 QPS. After batch aggregation, reporting QPS is significantly lower than the internal message QPS. Using the game package name as the Redis key avoids hotspot contention.

Applicable Scenarios

The design solves scenarios that require single‑producer, batch‑consumer messaging, supports multiple logical topics via multi‑key usage, and can be downgraded to a FIFO fixed‑length log queue. All operations run in O(N) time complexity.

Conclusion

The article demonstrates a Redis‑based pseudo‑message queue built with native commands and Lua scripts, achieving message grouping, fixed‑length queues, and batch consumption in a production environment.