Why Kafka Beats Redis List: A Deep Dive into Modern Message Queue Architecture

Basic Queue Concept

A simple in‑memory queue can be implemented with a doubly linked list, providing push_front (add to head) and pop_tail (remove from tail). Producers push messages to the head, consumers pop from the tail.

While easy to build, such a structure struggles under massive concurrent reads and writes.

Redis as a Queue

Redis offers the list data type, which directly maps to the queue abstraction: LPUSH adds to the left, RPOP removes from the right. Redis’s internal optimizations make it faster than a hand‑rolled list.

However, Redis lists have several critical shortcomings:

Persistence : AOF and RDB are auxiliary and not fully reliable; a crash can lose data.

Hot‑key bottleneck : All operations on a single list hit one Redis instance, limiting horizontal scaling.

No ACK mechanism : Once RPOP removes a message, failed consumer processing cannot be recovered.

Single consumer : Only one consumer can read a message; multiple services cannot share the same stream.

No re‑consumption : After a consumer crashes, the message is gone and cannot be replayed.

Some of these issues can be mitigated by using RocksDB/LevelDB‑compatible stores that speak the Redis protocol, but they still cannot solve hot‑key and multi‑consumer problems.

Redis 5.0 introduced STREAM, borrowing ideas from Kafka, yet it still falls short of a full‑featured message broker.

Kafka Architecture

Kafka was designed to address the limitations above. Its core concepts:

Partition : A topic is split into multiple partitions, each stored on a different broker, distributing load.

Append‑only log : New messages are appended to the end of a partition log; a cursor (offset) tracks consumption without deleting data.

ACK and consumer groups : Consumers commit offsets only after successful processing, enabling reliable ACK semantics and replay.

Consumer groups : Each group has its own cursor, allowing multiple independent consumers (broadcast) while a single partition is consumed by only one consumer within a group.

Kafka stores each partition as a series of segment files. When a segment reaches a size limit (e.g., 100 MiB), a new segment is created. This simplifies retention: whole expired segments can be deleted.

To locate a message by offset, Kafka uses a two‑step process:

Binary search the segment filenames (named by the first offset in the segment) to find the containing segment.

Read the corresponding .index file, which maps offsets to byte positions inside the segment.

Because storing an index entry for every message would be costly, Kafka employs a sparse index: only every Nth message (e.g., every 10) is recorded. A binary search finds the nearest entry, then the log is scanned forward a few records to reach the exact offset.

Kafka High Availability

Each partition has a leader and one or more followers. Writes go to the leader; followers replicate the log. Replication guarantees durability and enables automatic failover: if the leader crashes, a synchronized follower is elected as the new leader.

Acknowledgment strategies:

Leader‑only ACK : Fast, high throughput, but risk of data loss if the leader fails before followers sync.

All‑replicas ACK : Safer, but adds latency and can cause producer timeouts.

These settings can be tuned per application.

Kafka Pros and Cons

High performance (up to 1 M TPS per broker).

Low latency.

Strong availability via replication and ISR.

Mature tooling and ecosystem (e.g., Kafka Streams).

Limitations: scaling partitions requires careful rebalancing; adding brokers does not automatically relieve hot‑key pressure; consumer group rebalance can cause duplicate consumption.

Pulsar Overview

Pulsar separates compute and storage. The stateless broker handles client requests, while Apache BookKeeper provides durable segment storage.

Key differences from Kafka:

Broker is stateless, making horizontal scaling trivial.

Segments are written to BookKeeper ledgers, which are replicated across BookKeeper nodes (bookies). This enables fine‑grained storage scaling without moving whole partitions.

When a broker fails, another broker can take over the partition without data migration because the data resides in BookKeeper.

BookKeeper writes each segment to a configurable number of bookies (replication factor) and requires a configurable number of acknowledgments before confirming a write.

Pulsar Consumption Models

Pulsar introduces the concept of subscription , analogous to Kafka consumer groups but with four distinct modes:

Exclusive : Only one consumer may attach to the subscription.

Failover : Multiple consumers can attach, but only one receives messages; if it fails, another takes over.

Shared : Messages are round‑robin distributed among all consumers.

Key‑shared : Messages with the same key are routed to the same consumer, preserving ordering per key.

These models support both queue‑style and stream‑style processing.

Storage‑Compute Separation Benefits

By offloading durable storage to BookKeeper, Pulsar can add storage capacity simply by adding more bookies. Segments are evenly distributed, avoiding the monolithic log file growth seen in Kafka and simplifying expansion.

Overall, both Kafka and Pulsar solve the fundamental challenges of distributed messaging—scalable ingestion, reliable persistence, and flexible consumption—through different architectural choices: Kafka embeds storage in the broker, while Pulsar externalizes it to a dedicated log service.