Design and Implementation of a Redis‑Based Pseudo Message Queue for Game Behavior Data

This document presents a technical solution developed by the Vivo Internet Server team for collecting and reporting game‑level behavior data. The business requirements include batch reporting of up to 128 behaviors per game, handling large data volumes with possible data loss, and enforcing a strict timeliness constraint that only data generated within the last three minutes may be reported.

The problem is modeled as a classic producer‑consumer scenario. Within a JVM process, the team considered using a queue with locks or a lock‑free Disruptor; across processes, they evaluated message brokers such as RocketMQ or Kafka. However, neither approach satisfied the need for game‑dimension partitioning, batch processing, and strict timeliness.

Two concrete solutions were compared:

Solution 1: Use a message queue (RocketMQ/Kafka) to store reports. The consumer would need to aggregate messages by game dimension, but the queue itself does not address the required partitioning, batching, or timeliness.

Solution 2: Build a pseudo‑message middleware on Redis. A Redis list is used as a fixed‑length queue per game, LRANGE provides batch consumption, and multithreaded processing on the business side ensures timeliness. This approach meets all three constraints.

After comparison, the team adopted the Redis‑based design, which consists of:

Using a list per game to store behavior messages (the list key is the game package name).

Maintaining a global list that holds the set of games that currently have messages.

Employing a set to guarantee uniqueness of game entries.

Production Process

Push a behavior record into the game‑specific queue.

If the game is already present in the global set, return; otherwise, add the game to the set.

Consumption Process

Iterate over the global game list and select a game.

Use LRANGE on the game's queue to fetch a batch of messages, then LTRIM to remove the consumed items.

Technical Principles

The solution relies on Redis core commands combined with Lua scripts to guarantee atomicity:

Lua scripts execute a sequence of commands (e.g., LLEN , RPOP , LPUSH ) as a single atomic operation.

Fixed‑length queues are implemented by checking LLEN and performing LPOP before RPUSH when the maximum size is reached.

Batch consumption uses LRANGE to retrieve a range of elements and LTRIM to keep the remaining tail.

Key Redis commands used:

LLEN key                // O(1) length of list
LPOP key [count]       // Remove from left, O(N)
RPUSH key element ...  // Append to right, O(N)
LRANGE key start end   // O(S+N) range query
LTRIM key start stop   // O(N) trim operation
SADD key member ...    // O(1) add to set
SISMEMBER key member   // O(1) membership test

Sample Lua script for producing messages (fixed‑length queue):

EVAL script numkeys key [key ...] arg [arg ...]
-- The script checks LLEN, performs LPOP if needed, then RPUSH and sets expiration atomically.

Sample Lua script for consuming messages (batch pull):

EVAL script numkeys key [key ...] arg [arg ...]
-- The script runs LRANGE to fetch a batch and LTRIM to remove them, all atomically.

Technical Application

Production and consumption are driven by the Lua scripts, ensuring atomic updates. The system adopts a pull‑based consumption model with multiple threads, which proved to be more cost‑effective than a push model based on Redis Pub/Sub.

Online Performance

Deployed in production, the pipeline handles roughly 10,000 QPS for both production and consumption. Data is sharded by game package name, avoiding hotspot issues.

Applicable Scenarios

The approach is suitable for any scenario that requires single‑producer, batch‑consumer messaging with per‑topic (game) isolation, low latency, and O(N) batch processing. It can also serve as a FIFO fixed‑length log queue.

Conclusion

The paper demonstrates how Redis native commands combined with Lua scripting can emulate a lightweight message queue, providing game‑dimension partitioning, fixed‑length queues, and batch consumption with atomic guarantees. The solution has been successfully rolled out in a live environment, offering a reusable pattern for similar high‑throughput, low‑latency use cases.