Spring Boot Redis Stream Reliable Message Queue: Persistence, Consumer Groups & Stream Processing

1. Scenario and Selection

The author explains the motivation: order events required asynchronous processing (notifications, inventory deduction, points update, reporting). Synchronous calls caused high latency; direct service calls created tight coupling. Introducing a full Kafka or RabbitMQ cluster added operational overhead because the team already maintained Redis. Redis 5.0's Stream data type proved sufficient for medium-scale workloads (up to ~100k messages/second on a single instance). For very high throughput (tens of millions per day), Kafka remains the better choice.

2. Redis Stream Core Concepts

Redis Stream is a log-structured data type. Internally it uses listpack for message storage and a radix tree (rax) for ID indexing, enabling efficient range queries. Each message has an auto-incrementing ID like 1700000000000-0. Key concepts:

Entry : A message consisting of an ID and field-value pairs.

Consumer Group : Allows multiple independent consumer groups on the same stream; within a group each message is delivered to only one consumer.

PEL (Pending Entries List) : Tracks unacknowledged messages per consumer. A message enters the consumer's PEL after XREADGROUP and is removed only after explicit XACK.

last_delivered_id : The last message ID delivered to the consumer group.

XREADGROUP : Command to read messages as part of a consumer group; > reads only new messages, while a specific ID reads from the PEL.

XACK : Acknowledges successful processing.

The PEL is the cornerstone of reliability; without it Stream would be no different from simple pub/sub.

3. Spring Boot Integration: StreamTemplate

3.1 Dependencies and Configuration

Spring Boot 2.7+ uses spring-boot-starter-data-redis which includes Spring Data Redis 3.x with full Stream support. Standard application.yml configuration for a standalone Redis instance:

spring:
  data:
    redis:
      host: localhost
      port: 6379
      timeout: 5s
      lettuce:
        pool:
          max-active: 16
          max-idle: 8

3.2 StreamTemplate Implementation

A wrapper around StringRedisTemplate.opsForStream() to simplify send, consume, acknowledge, and retry operations. Key methods: send(key, map) – uses XADD, returns message ID. createGroup(key, group) – creates consumer group; ignores BUSYGROUP error if group exists. Defaults to reading from latest ( $); use ReadOffset.from("0") to consume from head. readNew(key, group, consumer, count) – reads new messages via XREADGROUP ... > with blocking (5 seconds). readPending(key, group, consumer, count) – reads unacknowledged messages from PEL using ReadOffset.from("0"). ack(key, group, recordIds) – sends XACK. pendingSummary(key, group) – returns PendingSummary for monitoring.

Important pitfalls: ReadOffset.latest() maps to > (new messages only); ReadOffset.from("0") reads from PEL. lastConsumed() is for positioning, not for reading new messages. createGroup defaults to $ (latest); to consume from beginning pass ReadOffset.from("0").

In Redis Cluster, a Stream's key is hashed to a single slot; consumer group state follows the key. Cross-node consumer groups are not supported, so horizontal scaling requires manual sharding (multiple Stream keys) or migrating to Kafka.

3.3 Publishing Order Events

Example publisher builds a map with fields eventId (UUID), type, orderId, userId, amount, timestamp and sends to stream order:events. Redis auto-generates ordered IDs.

3.4 Consuming Messages: Scheduled Polling + Manual ACK

A @Scheduled task (fixedDelay 1000ms) polls new messages. Consumer name combines hostname and a random long to ensure uniqueness across restarts. For each record:

Call business handler orderEventHandler.handle(record.getValue()).

If handler returns true, acknowledge with streamTemplate.ack().

If handler returns false (business failure), send to dead-letter queue then acknowledge to avoid blocking PEL.

If exception occurs, log error and do not acknowledge ; message stays in PEL for later retry.

Only after XACK does Redis consider the message processed. Unacknowledged messages remain in PEL and can be reclaimed via XAUTOCLAIM or manual PEL reads.

4. Consumer Group Competition and Load Balancing

Multiple consumer groups consume the same stream independently. Within a group, multiple consumers compete for messages. Redis maintains a last_delivered_id and distributes new messages sequentially to waiting consumers (not strict round-robin). This is described as "competition" rather than static partitioning like Kafka. Adding consumer instances increases throughput. However, Redis does not rebalance load if a consumer is slow; tasks should have similar latency or be kept lightweight.

5. Message Idempotency and Deduplication

During consumer failover, XAUTOCLAIM may reassign a message to another consumer, causing duplicate processing. Business logic must be idempotent. Two-layer approach:

Redis Set pre-filter : Check processed:event:ids set; if present, skip. Risk: entries may be evicted.

Database unique key as ultimate guard : Insert into order_event table with event_id as primary key. On DuplicateKeyException, treat as already processed and return true.

Code example shows @Transactional method catching DuplicateKeyException. Combining both layers yields best reliability.

6. Reliability: PEL, XAUTOCLAIM, and Dead Letter Queue

6.1 PEL Role

After a consumer reads a message, its ID enters that consumer's PEL (stored in Redis). If the consumer crashes, the message persists in PEL until reclaimed.

6.2 Consumer Restart: Process PEL First

On restart, call readPending (ID=0) to re-process unacknowledged messages. Business logic must be idempotent because partial processing may have occurred before crash.

6.3 XAUTOCLAIM: Automatic Timeout Reclamation

Redis 6.2 introduced XAUTOCLAIM to claim messages idle longer than a threshold, incrementing delivery count. Spring Data Redis wrapper:

public List<MapRecord<String, Object, Object>> autoClaim(String key, String group, String consumer, Duration minIdleTime, long count) {
  return redisTemplate.opsForStream().autoClaim(
    key, group, consumer,
    ClaimOptions.minIdle(minIdleTime),
    StreamReadOptions.empty().count(count)
  ).getRecords();
}

A scheduled job (cron every minute) claims messages idle >5 minutes, processes them, and acknowledges.

6.4 Dead Letter Queue (DLQ)

Redis Stream has no built-in DLQ; create a separate stream (e.g., order:events:dead-letter). Track delivery attempts with a Redis hash key attempt:{messageId}. Increment on each retry; if attempts exceed threshold (e.g., 5), send to DLQ, acknowledge original message, and delete attempt counter.

7. Consumer Failures and Cluster Reliability

7.1 Consumer Instance Crash

Other consumers in the group will claim orphaned PEL messages via XAUTOCLAIM. Requires stateless, idempotent business logic.

7.2 Redis Master-Slave Failover

With Sentinel or Cluster, Stream data and consumer group state replicate to replicas. On master failure, replica promotes. Data durability depends on AOF configuration: appendfsync everysec (max 1 second loss) or always (no loss, performance impact). Lettuce client auto-detects topology changes; application should implement retry logic.

7.3 Redis Cluster Limitations

A Stream maps to a single hash slot, so all its messages reside on one node. No native partitioning across nodes like Kafka. Horizontal scaling requires manual sharding (e.g., order:events:0, order:events:1) with client-side routing. If data volume exceeds single-node capacity, migrate to Kafka.

7.4 Graceful Shutdown

In @PreDestroy, set a running flag to false, allowing the consumption loop to exit naturally. Unfinished messages remain unacknowledged in PEL; other consumers will claim them via XAUTOCLAIM, achieving near-zero loss.

8. Comparison with RabbitMQ and Kafka

Redis Stream : Lightweight, lowest ops cost, sufficient performance (~100k msg/s). Lacks built-in DLQ, delayed queues; messages stored in memory (though persisted via AOF/RDB), so memory cost higher. Best for moderate volume, teams already running Redis.

RabbitMQ : Mature, flexible routing, built-in DLQ and delayed queues, high reliability. Throughput limited (~10k msg/s). Operational complexity higher (broker, vhosts, mirrored queues). Good for enterprise integration.

Kafka : Million msg/s throughput, partitioned ordering, message replay, stream processing. High ops overhead (ZooKeeper/KRaft, tuning). Overkill for non-big-data scenarios.

Rule of thumb: if you already have Redis and message volume is modest, use Stream; don't add another "zoo" of middleware.

9. Monitoring and Capacity Planning

Key metrics to watch:

Message backlog : XLEN on the stream; rising backlog indicates consumers falling behind.

Consumer group state : XINFO GROUPS shows unacknowledged count (PEL size) per group.

Consumer state : XINFO CONSUMERS reveals per-consumer PEL size and idle time; long idle suggests a dead consumer.

Redis health : Memory usage, AOF rewrite frequency. Set maxmemory-policy noeviction to prevent message eviction.

Prometheus alerts via redis_exporter, e.g., alert when redis_stream_consumers_lag > 1000 for 5 minutes. Capacity estimate: ~200 bytes/message; 10M messages/day ≈ 2 GB data + PEL metadata ≈ 3 GB. If memory pressure grows, consider message compression or migrating to Kafka.

10. Closing Principles

Three core principles for production use:

Business must be idempotent – never assume "no duplicates".

Enable AOF persistence – don't rely on Redis speed alone for durability.

Monitor PEL backlog and consumer idle time – set alerts before issues explode.

The provided code snippets are sufficient to get started; further questions welcome in comments.