How to Build a High‑Performance Game Event Queue with Redis and Lua

Business Background

In a game‑centric data collection scenario, the system must gather up to 128 distinct behavior events per game, support batch reporting, tolerate partial data loss, and ensure that only events from the last three minutes are reported.

Technical Selection

The problem maps to a classic producer‑consumer model. Within a JVM process, queues can be implemented with locks or the lock‑free Disruptor; across processes, MQ solutions like RocketMQ or Kafka are typical. However, these do not satisfy the required per‑game dimension grouping, batch size, and strict timeliness.

Option 1 : Use a message queue (RocketMQ/Kafka) and aggregate on the consumer side. This fails to meet the per‑game split, batch, and latency requirements.

Option 2 : Build a pseudo‑message middleware using Redis. Redis lists provide queue semantics, and Lua scripts guarantee atomic operations. This approach can enforce fixed‑length queues, batch consumption via LRANGE, and multithreaded processing to meet timeliness.

Solution Comparison

Store per‑game behavior messages in a fixed‑length List keyed by the game identifier.

Maintain a global List of games that currently have pending messages.

Use a Set to ensure the global game list contains unique entries.

Production Process

Step 1: Push a behavior event into the game‑specific queue.

Step 2: If the game is already in the global Set, return; otherwise, proceed.

Step 3: Add the game identifier to the global game list.

Consumption Process

Step 1: Iterate over the global game list to select a game.

Step 2: Use LRANGE on the game’s list to fetch a batch of events, then process them.

Technical Principles

Redis commands combined with Lua scripts implement the entire workflow. The producer uses LLEN, LPOP, and RPUSH to maintain a fixed‑length queue; the consumer uses LRANGE and LTRIM for batch consumption. All operations are atomic, ensuring thread safety and O(N) time complexity where N is the batch size.

Lua Script Example

EVAL script numkeys key [key ...] arg [arg ...]</code><code>    // Time complexity depends on the script itself</code><code>    > eval "return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}" 2 key1 key2 first second</code><code>1) "key1"</code><code>2) "key2"</code><code>3) "first"</code><code>4) "second"</code><code>Redis uses the same Lua interpreter to run all the commands.</code><code>Scripts execute atomically, similar to MULTI/EXEC.

Key Lua capabilities:

Atomic execution of multiple commands guarantees thread safety.

Combines List commands to implement fixed‑length queues and batch consumption.

Supports only single‑key operations; multi‑key scripts may cause redirection errors in cluster mode.

List Commands

LLEN key                // O(1) length of the list</code><code>LPOP key [count]       // O(N) remove N elements from the left</code><code>RPUSH key element ...   // O(N) push N elements to the right

By chaining LLEN, RPUSH, and LPOP, a bounded queue is realized: when the length reaches the limit, the oldest element is popped before pushing a new one.

LRANGE / LTRIM

LRANGE key start stop   // O(S+N) return N elements starting at offset S</code><code>LTRIM key start stop   // O(N) keep only the specified range

These commands enable batch retrieval and removal of processed events.

Set Commands

SADD key member ...     // O(1) add members</code><code>SISMEMBER key member   // O(1) check existence

The Set ensures each game appears only once in the global list.

Technical Application

Producing Messages

DEFINE LUA SCRIPT</code><code>CACHE_NPPA_EVENT_LUA =</code><code>    "local retVal = 0 " +</code><code>    "local key = KEYS[1] " +</code><code>    "local num = tonumber(ARGV[1]) " +</code><code>    "local val = ARGV[2] " +</code><code>    "local expire = tonumber(ARGV[3]) " +</code><code>    "if (redis.call('llen', key) < num) then redis.call('rpush', key, val) " +</code><code>    "else redis.call('lpop', key) redis.call('rpush', key, val) retVal = 1 end " +</code><code>    "redis.call('expire', key, expire) return retVal";</code><code>-- Execute script</code><code>String data = JSON.toJSONString(nppaBehavior);</code><code>Long retVal = (Long)jedisClusterTemplate.eval(CACHE_NPPA_EVENT_LUA, 1, NPPA_PREFIX + nppaBehavior.getGamePackage(), String.valueOf(MAX_GAME_EVENT_PER_GAME), data, String.valueOf(NPPA_TTL_MINUTE * 60));

Combines LLEN, RPUSH, and LPOP to enforce a fixed‑length queue.

Lua guarantees atomic execution of these commands.

Consuming Messages

DEFINE LUA SCRIPT</code><code>QUERY_NPPA_EVENT_LUA =</code><code>    "local data = {} " +</code><code>    "local key = KEYS[1] " +</code><code>    "local num = tonumber(ARGV[1]) " +</code><code>    "data = redis.call('lrange', key, 0, num) redis.call('ltrim', key, num+1, -1) return data";</code><code>-- Execute script</code><code>Integer batchSize = NppaConfigUtils.getInteger("nppa.report.batch.size", 1);</code><code>Object result = jedisClusterTemplate.eval(QUERY_NPPA_EVENT_LUA, 1, NPPA_PREFIX + gamePackage, String.valueOf(batchSize));

Uses LRANGE + LTRIM for batch consumption.

Lua ensures the two commands run atomically.

Important Considerations

In Redis cluster mode, Lua scripts should operate on a single key; multi‑key scripts cause redirection errors.

Different Redis versions handle null returns from Lua differently; consult the official docs.

Hot games receive dedicated consumer threads to prioritize processing based on configured priorities.

Online Performance

Production and consumption QPS reach roughly 10 k QPS. After batch reporting, the upstream reporting QPS is far lower than the internal message flow. Using the game package as the key avoids hotspot contention.

Applicable Scenarios

The Redis‑based pseudo‑MQ solves scenarios requiring single‑producer, batch‑consumer messaging, supports multi‑topic patterns via multiple keys, and can be downgraded to a FIFO fixed‑length log queue. It delivers O(N) batch consumption while maintaining low latency.

Conclusion

This article demonstrates a practical, Redis‑native approach to building a lightweight message queue for game event reporting. By leveraging Lua scripts and core Redis data structures, it achieves message grouping, fixed‑length queues, and efficient batch consumption, and has been validated in production with stable performance.