Kafka and RocketMQ Architecture: Availability, Reliability, and Design Considerations

The article begins by presenting the overall architecture of Kafka, highlighting its main components: Producer, Consumer, Kafka cluster, and ZooKeeper, which acts as a NameServer storing metadata and handling leader election.

It explains that ZooKeeper provides high availability, while the Broker stores messages. The discussion then moves to Kafka's availability, noting that its reliance on ZooKeeper does not affect cluster availability due to ZooKeeper's own fault‑tolerance.

Kafka achieves high availability through a replication strategy: each partition has multiple replicas distributed across different brokers, with one replica acting as the leader. Writes go to the leader and are replicated to followers, allowing the system to remain operational when a broker fails.

Reliability is ensured by persisting messages to disk (synchronous or asynchronous flush) and replicating them to other nodes, guaranteeing that messages are not lost unless all nodes fail simultaneously.

An evaluation lists Kafka's advantages—reduced functionality due to ZooKeeper handling coordination and high machine utilization—and disadvantages, such as added external dependency and complexity of implementing the replication mechanism.

The article then introduces RocketMQ, describing its components: Producer, Consumer, NameServer, and Broker. Unlike Kafka, RocketMQ implements a lightweight, stateless NameServer and uses a master‑slave model for broker availability.

RocketMQ's availability is achieved through multiple master brokers for a topic and a master‑slave replication that can fail over to the slave. Its reliability relies on synchronous disk flushing and master‑slave replication, with optional synchronous double‑write to mitigate data loss.

The evaluation of RocketMQ points out its strengths—no external dependencies and simpler deployment—and weaknesses, such as low machine utilization due to the master‑slave architecture.

Several hybrid and alternative architectures are explored, including combining Kafka's replication with RocketMQ's NameServer approach, removing the NameServer altogether, and using gossip protocols for metadata replication.

The proposed simplified MQ design adopts a single‑node NameServer, master‑slave brokers, and stores metadata in brokers, aiming for high availability and reliability while keeping the implementation straightforward.

In conclusion, the article summarizes the comparative analysis of Kafka and RocketMQ, presents the author's reflections on MQ design trade‑offs, and outlines a baseline architecture for future detailed discussions.