RabbitMQ vs Kafka: Architectural Comparison and Selection Guidance

Hello everyone, I am a top architect.

Introduction

Open‑source community offers many excellent queue middleware such as RabbitMQ and Kafka. Each queue seems to have its own characteristics, making it confusing to choose the right one for a project. This article examines whether to select RabbitMQ or Kafka.

RabbitMQ Architecture

RabbitMQ is a distributed system with several abstract concepts.

broker : the service program running on each node, responsible for maintaining queues on that node and forwarding queue operation requests.

master queue : each queue consists of a master queue and several mirror queues.

mirror queue : a backup of the master queue. If the node hosting the master queue fails, a mirror queue is promoted to master. Mirror queues only replicate data and are not intended to handle client read/write load.

The diagram shows a cluster with two nodes, each running a broker. Each broker maintains the queues on its own machine and brokers can communicate with each other. The cluster contains two queues (A and B), each split into a master queue and a mirror queue.

Queue Consumption

Two consumers consume from queue A, each connected to a different machine in the cluster. Any node in the RabbitMQ cluster holds metadata for all queues, so a client can connect to any node. The difference is whether the consumer is connected to the node hosting the master queue or a node hosting only a mirror queue.

Because a mirror queue must stay consistent with its master, all read/write operations must go through the master queue; the master then synchronizes changes to the mirror. Even if a consumer connects to a non‑master node, its operations are routed to the master node before being processed.

Queue Production

The production principle is the same as consumption: if a producer connects to a non‑master node, the request is routed to the master queue.

This design reveals a limitation of RabbitMQ: the master queue resides on a single node, creating a performance bottleneck and limiting throughput. Although RabbitMQ is implemented in Erlang for efficiency, it cannot escape this architectural constraint.

Kafka

Kafka was essentially designed to address the bottleneck of RabbitMQ by turning a single master queue into multiple masters, i.e., multiple partitions. If one machine cannot handle the QPS, the load is spread across many machines. Each partition (master queue) holds a distinct subset of messages, and there is no overlap between partitions.

In Kafka, each master queue is called a Partition . A topic may have many partitions, each with several replica partitions that synchronize similarly to RabbitMQ’s mirror queues.

Assuming a single topic, producers randomly send messages to a partition, and the partition then synchronizes the data to its replicas.

When consuming, Kafka introduces the concept of a Consumer Group . Within a group, each message from a topic is delivered to only one consumer, ensuring that consumers in the same group process different messages. Different groups can share the same topic, effectively creating multiple logical copies of the queue.

Kafka retains messages for a configurable period instead of deleting them immediately after consumption. This retention allows all consumer groups to see the same data within the retention window, making Kafka suitable as a company‑wide data bus.

Consumers read from the partition’s leader (master) and, for performance, each consumer is typically mapped one‑to‑one with a partition. If there are more consumers than partitions, some consumers will receive no messages.

Consequently, Kafka is designed for high throughput: configuring 100 partitions can distribute the load across 100 machines, far surpassing RabbitMQ’s single‑master limitation.

Conclusion

This article only compares Kafka and RabbitMQ, though many other open‑source queues exist (ZeroMQ, RocketMQ, JMQ, etc.). Based on throughput requirements, the key differences are:

Low throughput: both Kafka and RabbitMQ are suitable.

High throughput: Kafka is the better choice.

The information is drawn from the official RabbitMQ and Kafka documentation; for a deeper understanding, consult the original docs and perform your own architectural comparisons.

Source: cnblogs.com/haolujun/p/9632835.html