RabbitMQ vs Kafka: Architectural Comparison and Performance Considerations

Introduction

Open‑source queue middleware such as RabbitMQ and Kafka each have distinct characteristics, making it difficult to choose the right one for a project.

RabbitMQ Architecture

RabbitMQ is a distributed system with several key concepts:

broker : the service program running on each node that maintains queues and forwards requests.

master queue : each queue has a primary instance.

mirror queue : a backup of the master queue; if the master node fails, the mirror is promoted, but mirrors are not meant to handle client read/write load.

Queue Consumption

Consumers can connect to any node in the cluster because each node holds metadata for all queues. However, all read/write operations must go through the master queue, and the master synchronizes changes to its mirrors.

Queue Production

Production follows the same principle: if a producer connects to a non‑master node, the request is routed to the master queue.

Because RabbitMQ relies on a single master queue per queue, its throughput is limited, representing a performance bottleneck despite its Erlang implementation.

Kafka

Kafka addresses RabbitMQ's limitation by replacing the single master queue with multiple masters, called partitions . Each partition handles a subset of the traffic, allowing the load to be spread across many machines.

Each partition has replica partitions for backup, with a synchronization mechanism similar to RabbitMQ.

Message Consumption

Kafka introduces the concept of a consumer group . Within a group, each consumer receives a distinct subset of messages from a topic, while different groups can share the same topic data.

Messages are retained for a configurable period, enabling all groups to read the same data within that window, which makes Kafka suitable as a data bus.

Message Production

Producers randomly send messages to any partition; the partition then replicates the data to its replicas.

Consumers read from the leader partition, and if there are more consumers than partitions, some consumers will not receive messages.

Kafka’s design allows high throughput; for example, 100 partitions can distribute load across 100 machines, outperforming RabbitMQ’s single‑node limitation.

Conclusion

The article compares RabbitMQ and Kafka, noting that while both can handle low throughput, Kafka excels at high‑throughput scenarios. Selecting a queue should be based on architectural differences and specific requirements such as throughput.