Can Redis Replace Kafka? Comparing List, Pub/Sub, and Stream Queues

Redis List as a Queue

Redis List is essentially a doubly linked list where head and tail operations have O(1) complexity, making it suitable for simple FIFO queues.

Typical usage:

127.0.0.1:6379> LPUSH queue msg1
(integer) 1
127.0.0.1:6379> LPUSH queue msg2
(integer) 2

Consumers pull messages with RPOP:

127.0.0.1:6379> RPOP queue
"msg1"
127.0.0.1:6379> RPOP queue
"msg2"

Problem 1: CPU spin‑wait – Continuously calling RPOP in a loop wastes CPU when the queue is empty. A common workaround is to sleep when no message is returned, but this adds latency.

Problem 2: No blocking read – Redis provides BLPOP / BRPOP for blocking reads, which pause the consumer until a new message arrives, eliminating spin‑wait and reducing latency.

No multi‑subscriber support: A message can be consumed by only one consumer.

Message loss on consumer failure: Once RPOP removes the item, a failed processing step cannot recover it.

Blocking read example:

while true:
    msg = redis.rpop("queue")
    // no message, continue loop
    if msg == null:
        continue
    // process message
    handle(msg)

Improved blocking read with BRPOP (timeout 0 means block indefinitely):

while true:
    msg = redis.brpop("queue", 0)
    if msg == null:
        continue
    handle(msg)

Publish/Subscribe Model (Pub/Sub)

Redis Pub/Sub is suitable for broadcast scenarios where multiple consumers need to receive the same message.

Example with two consumers subscribing to the same channel:

// 2 consumers subscribe the same channel
127.0.0.1:6379> SUBSCRIBE queue
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "queue"
3) (integer) 1

Publishing a message:

127.0.0.1:6379> PUBLISH queue msg1
(integer) 1

Both consumers receive the message:

127.0.0.1:6379> SUBSCRIBE queue
1) "message"
2) "queue"
3) "msg1"

Drawbacks of Pub/Sub:

If a consumer disconnects, the message is lost.

Redis does not guarantee persistence; a crash can discard in‑flight messages.

Message backlog can cause buffer overflow and kick consumers offline.

Therefore Pub/Sub is best for low‑reliability real‑time broadcasts (e.g., chat, notifications) rather than critical business flows.

Redis Streams – A More Mature Queue

Streams address many of the limitations of List and Pub/Sub. They use XADD to append messages and XREAD (or XREADGROUP) to consume them.

Producer adds messages ("*" lets Redis generate a unique ID):

127.0.0.1:6379> XADD queue * name zhangsan
"1618469123380-0"
127.0.0.1:6379> XADD queue * name lisi
"1618469127777-0"

Consumer reads the first five entries from the beginning:

127.0.0.1:6379> XREAD COUNT 5 STREAMS queue 0-0
1) 1) "queue"
   2) 1) "1618469123380-0"
      2) "name"
      3) "zhangsan"
   2) "1618469127777-0"
      2) "name"
      3) "lisi"

If no new messages exist, Redis returns (nil).

Blocking reads are possible by adding the BLOCK argument:

127.0.0.1:6379> XREAD COUNT 5 BLOCK 0 STREAMS queue 1618469127777-0

Streams also support consumer groups for parallel consumption. Create groups:

127.0.0.1:6379> XGROUP CREATE queue group1 0-0
OK
127.0.0.1:6379> XGROUP CREATE queue group2 0-0
OK

Each group can have its own consumer. Example for group1 :

127.0.0.1:6379> XREADGROUP GROUP group1 consumer COUNT 5 STREAMS queue >
1) 1) "queue"
   2) 1) "1618470740565-0"
      2) "name"
      3) "zhangsan"
   2) "1618470743793-0"
      2) "name"
      3) "lisi"

Both groups can read the same messages, achieving multi‑group consumption.

When a consumer processes a message, it must acknowledge it with XACK so the message is marked as done. If a consumer crashes before acknowledging, the message remains pending and can be re‑delivered when the consumer restarts:

127.0.0.1:6379> XACK queue group1 1618472043089-0

Pending messages can be fetched again with XREADGROUP using ID 0-0 :

127.0.0.1:6379> XREADGROUP GROUP group1 consumer1 COUNT 5 STREAMS queue 0-0

Streams are persisted to both RDB and AOF like any other Redis data type, so with proper persistence configuration they survive restarts.

Comparison with Professional Message Queues

Professional brokers (Kafka, RabbitMQ) guarantee two core properties:

No message loss

Message backlog handling

Producer reliability : Both Redis and professional queues can retry on network failures. In Redis, a failed LPUSH or XADD returns an error; the producer should retry up to a configured limit. Consumer reliability : Consumers must acknowledge messages (e.g., XACK for Streams, or transaction commits for Kafka/RabbitMQ). Without acknowledgment, the broker may redeliver the message. Broker reliability :

Redis AOF writes are asynchronous; a crash may lose recent writes.

Master‑slave replication is asynchronous, so a failover can lose data not yet replicated.

Kafka/RabbitMQ typically run as clusters with synchronous replication across nodes, reducing loss risk.

Message backlog in Redis consumes memory, risking OOM. Streams mitigate this by allowing a maximum length (e.g., XADD MAXLEN ). Kafka/RabbitMQ store data on disk, making large backlogs less problematic.

Conclusion

Redis can be used as a lightweight queue for simple scenarios, especially when operational overhead must be minimal. However, it has two major drawbacks:

Potential data loss under certain failure modes (AOF latency, async replication).

Memory‑bound backlog handling, which can lead to OOM.

For mission‑critical, high‑throughput, or durable messaging requirements, dedicated brokers like Kafka or RabbitMQ remain the safer choice.

References

https://redis.io/docs/latest/commands/

https://blog.csdn.net/zhangchb/article/details/120507983

https://redis.com/blog/redis-streams-a-practical-introduction/

https://kafka.apache.org/documentation/