Mastering Redis Streams: From Basics to Consumer Groups

Overview

Redis Streams, introduced in Redis 5.0, model a log‑like data structure that is append‑only and read‑only, offering powerful operations such as blocking reads and consumer groups, concepts originally popularized by Kafka.

Basic Concepts

Streams store entries as ID‑field/value pairs, similar to CSV rows. Adding entries uses XADD with * to auto‑generate a monotonically increasing ID composed of a millisecond timestamp and a sequence number.

> XADD mystream * sensor-id 1234 temperature 19.8

The XLEN command returns the number of entries.

> XLEN mystream
(integer) 1

Entry IDs

Each ID has the form <milliseconds>-<sequence>. The timestamp part is the local node time; if the clock moves backward, the previous timestamp is reused, preserving monotonicity. IDs enable efficient range queries.

Explicit IDs can be supplied instead of *, e.g., 0-1 or 1234-0.

> XADD mystream 0-1 field value
0-1

Fetching Data from Streams

Three main access patterns exist:

Range queries using XRANGE / XREVRANGE.

Non‑blocking reads with XREAD.

Blocking reads with XREAD + BLOCK or consumer groups.

Range Queries: XRANGE and XREVRANGE

Specify start and end IDs (or - and + for min/max). Optional COUNT limits results.

> XRANGE mystream - +
1) 1) 1518951480106-0
   2) 1) "sensor-id" "1234"
   3) 1) "temperature" "19.8"

Complexity is O(log(N)) to locate the start and O(M) to return M entries.

Using XREAD to Listen for New Elements

XREAD

can poll for new entries. Adding BLOCK 0 makes it wait indefinitely until new data arrives.

> XREAD BLOCK 0 STREAMS mystream $

Note: In the blocking form, the start and stop arguments are reversed compared to XREVRANGE .

Consumer Groups

Consumer groups allow multiple clients to share the workload of a single stream, ensuring each message is delivered to only one consumer within the group.

+----------------------------------------+
| consumer_group_name: mygroup            |
| last_delivered_id: 1292309234234-92   |
| consumers: "consumer-1" (pending 1292309234234-4) |
|            "consumer-42" (pending ...) |
+----------------------------------------+

Key guarantees:

Each message is processed by a single consumer.

Consumers are identified by a case‑sensitive name.

The group tracks the first unprocessed ID.

Consumers must explicitly acknowledge processed messages with XACK.

Pending messages are visible per consumer.

Creating a Consumer Group

Use XGROUP CREATE. The special ID $ starts the group at the current maximum ID, so only new messages are delivered.

> XGROUP CREATE mystream mygroup $
OK

Using MKSTREAM creates the stream if it does not exist.

> XGROUP CREATE newstream mygroup $ MKSTREAM
OK

Reading with XREADGROUP

Specify the group and consumer name. Use > as the ID to fetch only messages not yet delivered to any consumer.

> XREADGROUP GROUP mygroup Alice COUNT 1 STREAMS mystream >
1) 1) "mystream"
   2) 1) "message" "apple"

To read pending history, supply a numeric ID (e.g., 0).

> XREADGROUP GROUP mygroup Alice STREAMS mystream 0
1) 1) "mystream"
   2) 1) "message" "apple"

After processing, acknowledge with XACK.

> XACK mystream mygroup 1526569495631-0
(integer) 1

Pending Message Inspection and Claiming

XPENDING

shows total pending count, min/max IDs, and per‑consumer pending numbers.

> XPENDING mystream mygroup
1) 2
2) 1526569498055-0
3) 1526569506935-0
4) "Bob" 2

Use XCLAIM to transfer ownership of idle pending messages to another consumer.

> XCLAIM mystream mygroup Alice 3600000 1526569498055-0

Trimming (Capped Streams)

Limit stream length with XADD ... MAXLEN or XTRIM. The ~ modifier allows approximate trimming for better performance. > XADD mystream MAXLEN ~ 1000 * field value Trimming removes oldest entries once the limit is exceeded.

Special IDs

-

: Minimum possible ID. +: Maximum possible ID. $: Current maximum ID of the stream. >: In XREADGROUP, fetches messages never delivered to any consumer. *: In XADD, auto‑generates a new ID.

Persistence, Replication, and Safety

Streams and consumer‑group state are persisted to AOF/RDB and replicated to replicas. For strong durability, configure AOF with fsync always and consider WAIT to ensure writes reach replicas before proceeding.

Deleting Individual Entries

Use XDEL with an entry ID. Memory is reclaimed only when a whole radix‑tree node becomes empty.

> XDEL mystream 1526654999635-0
(integer) 1

Empty‑Length Streams

Unlike other data types, a stream is not automatically deleted when empty because it may hold consumer‑group metadata.

Latency Characteristics

Non‑blocking commands have typical Redis O(log(N)) or O(1) complexities. Blocking reads are push‑based: when XADD writes a new entry, waiting consumers are notified before the event loop returns, yielding sub‑millisecond latency in most cases.

Benchmark on a modest dual‑core instance with 10 k messages/sec and ten consumers showed 99.9 % of messages processed within 2 ms.