Disque: An Experimental Distributed In‑Memory Message Queue – Design and Usage Overview

Disque is an ongoing experimental distributed in‑memory message broker designed to capture the "Redis as a message queue" use case, offering a simple, high‑performance, non‑blocking C implementation similar to Redis while providing fault‑tolerant, scalable design.

As of January 2 2016 the project is in release‑candidate status, encouraging users to evaluate, report bugs, and share experiences, though it is not yet recommended for production.

What is a message queue? It enables programs to communicate by sending messages to named queues, where producers enqueue messages and consumers retrieve them, guaranteeing delivery even in the presence of failures.

Key features

Synchronously replicated task queue: new tasks are copied to W nodes before client acknowledgment; up to W‑1 nodes may fail without losing the message.

Supports both at‑least‑once and at‑most‑once delivery semantics, configurable per task via replication factor and retry time.

All nodes have identical roles (multi‑master); producers and consumers can connect to any node, with automatic load‑balanced message exchange.

High availability as an AP system: the queue remains operational as long as a single node is alive.

Asynchronous commands provide low‑latency operations with weaker delivery guarantees.

Explicit ACK mechanism lets consumers confirm successful processing; ACKs are replicated and eventually garbage‑collected.

Message ordering is based on task creation timestamps, though strict FIFO is not guaranteed.

Each task can be fine‑tuned with four parameters: replication factor, delay time, retry time, and expiration time.

Additional features include blocking queues, queue statistics, stateless iterators, visibility control commands, easy cluster scaling, and safe node removal without losing task replicas.

ACK and Retry

Disque uses client ACKs to avoid duplicate deliveries; ACKs are replicated across nodes and eventually cleaned up. If a node holding a replica is partitioned during acknowledgment, the message may be re‑queued when the partition heals.

Fast ACK (FASTACK)

FASTACK allows a client to send a single command that causes the node to discard the task and broadcast DELJOB to all nodes, reducing message traffic at the cost of reliability; if replicas are unreachable, the task may be re‑delivered.

Dead‑Letter Queue

Disque tracks two counters per task (NACK and additional retry) to decide when a message should be considered dead; workers can act on these counters by emailing, setting monitoring flags, moving to a special queue, or logging errors.

Disk Persistence

Although primarily in‑memory, Disque can persist tasks using an Append‑Only File (AOF) with configurable fsync strategies, allowing controlled restarts and upgrades while preserving task state.

Task ID

Each task receives a unique 40‑character ID prefixed with D- . Example:

D-dcb833cf-8YL1NT17e9+wsA/09NqxscQI-05a1

The ID components are:

Prefix D- .

First 8 bytes of the generating node’s ID ( dcb833cf ).

Base64‑encoded 144‑bit random part ( 8YL1NT17e9+wsA/09NqxscQI ).

TTL in minutes ( 05a1 ), encoding delivery guarantees.

The probability of ID collisions is astronomically low (≈2.2×10^49 possible IDs), making IDs effectively unique.

The article is a translation from the original GitHub repository https://github.com/antirez/disque and was first published by the High‑Availability Architecture community.