How to Use Redis Queues for High-Concurrency Log Processing

Message queues are commonly used to solve high‑concurrency problems through asynchronous processing. For example, log collection generates large volumes of data with high concurrent pressure; inserting directly into a database is unsuitable, but real‑time requirements are low, so logs can be cached in a queue and processed in batches.

Basic Idea

Insert log information into the queue for caching.

Periodically read the cache and batch‑insert into the database.

Implementation

Below is a simple pseudo‑code example.

1. Log Enqueue

Because the concurrency is high, the processing should be as simple as possible and can be exposed as an interface for log‑recording programs.

// Get log information
var info = getinfo();
// Add timestamp
info += "|" + time();
// Push into queue
redis.lpush("log", info);

2. Batch Insert Processing

Every minute retrieve N messages from the queue and insert them into the database in bulk. If an insertion error occurs, the failed messages are re‑queued for the next attempt.

// Number of messages to read
var count = N;
// Rollback array
var arr_rollback = array();

// Build SQL prefix
var sql = "insert into log (`content`, `createtime`) values ";

// Loop to read logs and build SQL
for(var i=0; i<count; i++){
    // Get message from queue
    var info = redis.rpop("log");
    // Stop if no more messages
    if(empty(info)){
        break;
    }
    // Save to rollback array
    arr_rollback.push(info);
    // Split info into content and timestamp
    var arr = split(info, "|");
    var loginfo = arr[0];
    var time = arr[1];
    // Append to SQL
    sql += " ('" + loginfo + "','" + time + "'),";
}

// Execute insert
sql = rtrim(sql, ",") + ";";
var result = mysql_query(sql);

// On failure, rollback
if(!result){
    foreach(arr_rollback as rec){
        redis.lpush("log", rec);
    }
}

3. Scheduling

The batch process can be triggered by a timer or a simple system cron job.