Mastering Redis 5.0 Streams: Data Structure, Commands, and Memory Optimizations

Redis 5.x Overview

Redis 5.x inherits all enhancements from 4.x and introduces new commands and features.

Stream Data Structure

The Stream type, added in Redis 5.0, is a persistent, multicast‑capable message queue.

It is implemented as a linked list of messages, forming a durable data structure (see Figure 1).

Key Characteristics of Streams

Multiple consumer groups can exist within a single Stream.

Each group maintains a Last_delivered_id pointing to the last consumed message.

Consumers in the same group share the Last_delivered_id; only one consumer can process a given message.

Each consumer tracks Pending_ids, which list messages sent to the client but not yet acknowledged.

Comparison with other Redis data structures is provided in Table 1.

Stream Commands Walkthrough

The typical workflow uses the following commands in order: XADD – add entries to a stream (optionally limit the maximum length). XGROUP – create a consumer group. XREADGROUP – read messages as a consumer. XACK – acknowledge successful processing of a message.

Message Acknowledgment

Unlike Pub/Sub, Streams support explicit acknowledgment. After a consumer reads a message with XREADGROUP or claims it with XCLAIM, the server cannot assume the message has been processed.

Therefore, once processing succeeds, the consumer must call XACK to clear the pending entry and free memory.

If a client fails to acknowledge (e.g., due to network issues), the pending entry remains. Re‑connecting clients should start reading from 0-0 to retrieve both pending and new messages, and they must be prepared for possible duplicate deliveries.

Memory Usage Optimizations

Active Fragmentation Defragmentation Frequent key updates cause memory fragmentation because Redis allocates new memory for changed values without releasing the old blocks to the OS. When used_memory_rss / used_memory exceeds 1.5, fragmentation is considered high. Redis 3.x required periodic restarts; Redis 4.x added automatic defragmentation and a MEMORY PURGE command; Redis 5.0 improves this further with an updated jemalloc version, offering faster, smarter, lower‑latency cleanup.

HyperLogLog Algorithm Optimization HyperLogLog provides cardinality estimation with minimal memory. Redis 5.0 refines the algorithm, reducing memory consumption for large‑scale counting (e.g., 1 KB vs. 1 MB for a B‑tree).

Enhanced INFO Output The INFO command now returns richer memory‑related statistics.

New and Optimized Commands

1. Client Management Enhancements

The redis-cli now includes built‑in cluster management (no need for the external redis-trib tool). Use redis-cli --cluster help for details.

Performance improvements for frequent connect/disconnect scenarios benefit short‑lived connections.

2. Simplified Sorted Set Operations

Two new commands were added: ZPOPMIN – removes and returns the lowest‑scoring members (optionally a count). ZPOPMAX – removes and returns the highest‑scoring members (optionally a count).

3. Expanded HELP Subcommand

The HELP subcommand now provides detailed usage examples, such as xinfo help for stream commands.

127.0.0.1:6379> xinfo help
1) XINFO <subcommand> arg ... arg. Subcommands are:
2) CONSUMERS <key> <groupname>  -- Show consumer groups of group <groupname>.
3) GROUPS <key>               -- Show the stream consumer groups.
4) STREAM <key>               -- Show information about the stream.
5) HELP                       -- Print this help.

4. Command‑Line Auto‑Completion redis-cli now displays parameter hints after a full command is typed, helping users remember syntax (e.g., typing zadd shows its arguments in a lighter color).

RDB Snapshot Supports LFU and LRU

Starting with Redis 5.0, RDB files store the eviction policy for each key, allowing the snapshot to preserve LFU (Least Frequently Used) and LRU (Least Recently Used) information in addition to FIFO.

FIFO – first‑in‑first‑out.

LRU – evicts keys that have not been used recently.

LFU – evicts keys with the lowest access frequency over a time window.