How to Build a High‑Performance Async Task Queue with Redis and Node.js

Problem Statement

Processing 100,000 string records sequentially at roughly 25 seconds each would require about 2.5 million seconds (≈29 days). An asynchronous task queue built with Redis and Node.js can reduce this runtime dramatically.

Architecture Overview

Each record is treated as an independent task stored in a Redis List. Multiple worker processes pop tasks concurrently, process them, and finish when the list becomes empty.

Deploy Redis with Docker

docker pull redis:latest
docker run -itd --name redis-local -p 6379:6379 redis

Redis is reachable at 127.0.0.1:6379. A GUI client such as Another Redis Desktop Manager can be used for inspection.

Node.js Redis Client (mqClient.ts)

import * as Redis from 'redis';
const client = Redis.createClient({ host: '127.0.0.1', port: 6379 });
export default client;

The client provides standard CRUD operations but returns results via callbacks.

Promise‑based Utility Wrapper (utils.ts)

import client from './mqClient';

export const getRedisValue = (key) =>
  new Promise(resolve => client.get(key, (err, reply) => resolve(reply)));

export const setRedisValue = (key, value) =>
  new Promise(resolve => client.set(key, value, resolve));

export const delRedisKey = (key) =>
  new Promise(resolve => client.del(key, resolve));

Creating Tasks (createTasks.ts)

import { TASK_NAME, TASK_AMOUNT, setRedisValue, delRedisKey } from './utils';
import client from './mqClient';

client.on('ready', async () => {
  await delRedisKey(TASK_NAME);
  for (let i = TASK_AMOUNT; i > 0; i--) {
    client.lpush(TASK_NAME, `task-${i}`);
  }
  client.lrange(TASK_NAME, 0, TASK_AMOUNT, (err, reply) => {
    if (err) { console.error(err); return; }
    console.log(reply);
    process.exit();
  });
});

This script pushes task-1 … task-20 into the Redis list identified by TASK_NAME (e.g., local_tasks).

Entry Point (index.ts)

import taskHandler from './tasksHandler';
import client from './mqClient';

client.on('connect', () => console.log('Redis is connected!'));
client.on('ready', async () => {
  console.log('Redis is ready!');
  await taskHandler();
});
client.on('error', e => console.log('Redis error! ' + e));

Task Handler (tasksHandler.ts)

import { popTask, setBeginTime, getRedisValue, setRedisValue, delRedisKey } from './utils';
import client from './mqClient';
import Redlock from 'redlock';
const redlock = new Redlock([client]);

async function handleTask(task) {
  return new Promise(resolve => {
    setTimeout(() => {
      console.log(`Handling task: ${task}...`);
      resolve();
    }, 2000); // simulate 2 s processing time
  });
}

export default async function tasksHandler() {
  // Record the start time of the first taken task
  await setBeginTime(redlock);

  const task = await popTask();
  await handleTask(task);

  // Increment completed‑task counter with a distributed lock
  try {
    const lock = await redlock.lock(`lock:${TASK_NAME}_CUR_INDEX`, 1000);
    let curIndex = Number(await getRedisValue(`${TASK_NAME}_CUR_INDEX`));
    await setRedisValue(`${TASK_NAME}_CUR_INDEX`, (curIndex + 1).toString());
    await lock.unlock().catch(() => {});
  } catch (e) { console.log(e); }

  // Continue processing recursively
  await tasksHandler();
}

Measuring Total Execution Time

Three Redis keys are used for timing: ${TASK_NAME}_SET_FIRST – flag indicating whether the first task has been taken. ${TASK_NAME}_BEGIN_TIME – timestamp (ms) when the first task was taken. ${TASK_NAME}_CUR_INDEX – counter of completed tasks.

When ${TASK_NAME}_CUR_INDEX equals the known total ( ${TASK_NAME}_TOTAL), the system logs the elapsed time and resets the markers.

export const setBeginTime = async (redlock) => {
  const lock = await redlock.lock(`lock:${TASK_NAME}_SET_FIRST`, 1000);
  const setFirst = await getRedisValue(`${TASK_NAME}_SET_FIRST`);
  if (setFirst !== 'true') {
    await setRedisValue(`${TASK_NAME}_SET_FIRST`, 'true');
    await setRedisValue(`${TASK_NAME}_BEGIN_TIME`, Date.now().toString());
  }
  await lock.unlock().catch(() => {});
};

Running the System

pm2 start ./dist/index.js -i 4 && pm2 logs

Four PM2 processes (matching an 8‑core CPU) run in parallel. With 20 tasks each taking 2 seconds, the total runtime is about 10 seconds.

References

Project repository: https://github.com/jrainlau/node-redis-missions-queue

PM2 cluster mode documentation: https://pm2.keymetrics.io/docs/usage/cluster-mode/

Another Redis Desktop Manager: https://github.com/qishibo/AnotherRedisDesktopManager

node‑redis package: https://www.npmjs.com/package/redis

node‑redlock package: https://www.npmjs.com/package/redlock