RabbitMQ vs Kafka: Architectural Comparison and Throughput Considerations

Open‑source community offers many excellent queue middlewares such as RabbitMQ and Kafka; each has distinct characteristics, making selection confusing for engineers.

RabbitMQ Architecture

RabbitMQ is a distributed system with several abstract concepts.

broker: the service program running on each node, maintaining queues and forwarding requests.

master queue: each queue has a primary queue and several mirror queues.

mirror queue: a backup of the master queue; when the master node fails, a mirror is promoted to master. Mirrors are not intended to handle client read/write load.

In a two‑node cluster, each node runs a broker; brokers communicate with each other. The cluster contains two queues (A and B), each split into a master queue and a mirror queue.

Queue Consumption

Consumers can connect to any node; the node they connect to may host the master queue or a mirror. All read/write operations must go through the master queue, which then synchronizes changes to its mirrors.

Queue Production

The production principle mirrors consumption: if a producer connects to a non‑master node, the request is routed to the master queue.

Because RabbitMQ relies on a single master queue per logical queue, its throughput is limited by that node.

Kafka

Kafka was designed to address RabbitMQ's limitation by replacing the single master with multiple masters, called partitions. Each partition acts as an independent master, and replicas provide backup.

In a Kafka cluster, a topic (the logical queue) is divided into multiple partitions; producers send messages to a random partition, which then replicates to its followers.

Consumers belong to a consumer group; each message in a topic is delivered to only one consumer within the same group, while different groups can read the same data. Kafka retains messages for a configurable period, enabling it to act as a data bus.

Reading also occurs from the partition leader, and if the number of consumers exceeds the number of partitions, some consumers will receive no messages.

Summary

The article only compares RabbitMQ and Kafka, though many other open‑source queues exist (ZeroMQ, RocketMQ, JMQ, etc.).

Low throughput requirements: both Kafka and RabbitMQ are suitable.

High throughput requirements: Kafka is preferable.

For a deeper understanding, consult the official documentation of each middleware and evaluate them against your specific use cases.