How to Choose Among Four Popular Message Queues?

01 Message Queue Basics

1.1 What Is a Message Queue?

A message queue is a container that stores messages during transmission, allowing one or multiple consumers to retrieve them. It consists of three elements:

Producer : generates and sends messages to the broker.

Broker : the processing center that stores, acknowledges, and retries messages, typically containing multiple queues.

Consumer : fetches messages from the broker for processing.

1.2 Message Queue Patterns

Point‑to‑point: multiple producers send to the same queue, but each message is consumed by only one consumer.

Publish/subscribe: a single message can be fetched and processed concurrently by multiple subscribers.

1.3 Application Scenarios

Decoupling : reduces coupling between services.

Asynchronous processing : allows delayed consumption.

Traffic shaping : buffers mismatched upstream/downstream capacities.

Log processing : e.g., Kafka for massive log transport.

Message communication : built‑in efficient communication mechanisms.

Broadcast : simplifies adding new downstream consumers.

02 Common Message Queues

ActiveMQ 5.x is receiving less community maintenance and is rarely used in large‑scale throughput scenarios, so the focus is on Kafka, RocketMQ, and RabbitMQ.

2.1 Kafka

Apache Kafka originated at LinkedIn as a distributed commit‑log system and later became an Apache project, often described as the "killer app for big data" in data collection, transmission, and storage.

It is a distributed platform that supports multiple partitions, replicas, and relies on Zookeeper for cluster coordination.

Important Concepts

Topic : a category of messages, analogous to a database table.

Partition : a topic can be split into several partitions that may reside on different machines, enabling scalability.

Batch : a group of messages written together for efficiency.

Consumer Group : a set of one or more consumers that share the consumption load.

Broker : an individual Kafka server that receives messages, assigns offsets, and persists them.

Broker Cluster : multiple brokers forming a cluster.

Rebalance : automatic redistribution of partitions among consumers when a consumer instance fails.

Kafka Architecture

A typical Kafka cluster includes producers, brokers, consumer groups, and a Zookeeper ensemble. Zookeeper manages configuration, leader election, and triggers rebalance when consumer groups change. Producers push messages to brokers; consumers pull messages from brokers.

Kafka Working Principle

After serialization, a message is assigned to a partition based on a strategy (round‑robin, random, or key hash). Messages with the same topic and partition are stored in the same batch and sent by a dedicated thread.

Partitions can be distributed across multiple machines, providing horizontal scalability. When a broker goes offline, its partitions are reduced accordingly.

Consumers within the same consumer group can consume multiple partitions, but a single partition is consumed by only one consumer in the group.

Adding consumers triggers a rebalance, redistributing partitions among the group.

Different consumer groups operate independently, allowing parallel consumption of the same partitions.

2.2 RocketMQ

RocketMQ is an open‑source Java‑based message middleware from Alibaba, offering high performance, reliability, real‑time capabilities, and suitability for large‑scale distributed systems.

Although inspired by Kafka, RocketMQ optimizes reliable transmission and transactional support. It is widely used within Alibaba for transactions, recharge, stream processing, message push, log streaming, and binlog distribution.

Important Concepts

NameServer : acts as a registry, similar to Zookeeper in Kafka.

Broker : a standalone RocketMQ server that stores messages and assigns offsets.

Topic : the primary message type; each message must belong to a topic.

Tag : a secondary type allowing differentiation within the same topic.

Group : can be a producer group or consumer group, subscribing to multiple topics.

Queue : comparable to Kafka's partition.

RocketMQ Working Principle

The model includes Producer Group, Topic, and Consumer Group. A topic contains multiple queues; producers place messages into queues using round‑robin selection.

Consumer groups and queues correspond to Kafka's consumer groups and partitions: a queue is consumed by only one consumer, while a consumer may consume multiple queues.

Offsets track the consumption position of each queue.

RocketMQ Architecture

Four core roles: NameServer, Broker, Producer, Consumer. The broker stores queues; a broker can host multiple topics, each with multiple queues.

For high‑volume topics, configure more queues and distribute them across different brokers to alleviate pressure. Uniform message volume across queues leads to balanced load; however, more queues on a single broker increase its load.

Brokers form a cluster with master/slave architecture. Slaves periodically sync data from the master (synchronously or asynchronously). If the master fails, the slave continues to serve consumption but cannot accept new writes.

The design closely mirrors Kafka's architecture.

2.3 RabbitMQ

Released in 2007, RabbitMQ is an open‑source message queue written in Erlang and implements the AMQP protocol.

AMQP emphasizes messaging, queuing, routing, reliability, and security, making it suitable for enterprise systems that prioritize data consistency and stability over raw performance.

Important Concepts

Channel : a virtual session for reading/writing messages; multiple channels can exist per client.

Exchange : receives messages and routes them to one or more queues based on routing rules.

RoutingKey : a key supplied by the producer to guide the exchange's routing decision.

Binding : a virtual link between an exchange and a queue, optionally containing routing keys.

RabbitMQ Working Principle

The AMQP model consists of three parts: producer, consumer, and server. The workflow is:

Producer connects to the server and opens a channel.

Producer declares an exchange and a queue, sets properties, and binds them using a routing key.

Consumer establishes a connection and opens a channel to receive messages.

Producer sends a message to the virtual host on the server.

The exchange routes the message to the appropriate queue based on the routing key.

Subscribed consumers retrieve and process the message.

Common Exchanges

RabbitMQ provides four exchange types: direct, topic, fanout, and headers. Detailed usage can be found on the official site.

官网入口：https://www.rabbitmq.com/getstarted.html

03 Message Queue Comparison

3.1 Kafka

Advantages:

High throughput and low latency (tens of thousands of messages per second, latency of a few milliseconds).

High scalability via multiple partitions per topic.

High stability with replicated data; the cluster continues operating if a node fails.

Durability, reliability, and replayability through persistent storage and backups.

Message ordering guarantees and exactly‑once consumption.

Rich third‑party management UI (e.g., Kafka‑Manager) and widespread adoption.

Disadvantages:

Single machine performance degrades sharply when exceeding ~64 partitions.

Lacks built‑in message routing, delayed delivery, and retry mechanisms.

Community updates are relatively slow.

3.2 RocketMQ

Advantages:

High throughput comparable to Kafka, capable of handling millions of messages per queue.

High scalability with a flexible distributed horizontal expansion architecture.

High fault tolerance via ACK mechanisms ensuring successful consumption.

Persistence and replay support.

FIFO ordering within a single queue.

Supports both publish/subscribe and point‑to‑point models, with pull and push delivery.

Provides Docker images and a feature‑rich dashboard for monitoring.

Disadvantages:

No built‑in message routing.

Limited client language support (primarily Java; C++ support is immature).

Ordering is only guaranteed when messages of the same type are hashed to the same queue.

Community activity is moderate.

3.3 RabbitMQ

Advantages:

Supports virtually all popular programming languages.

Rich routing capabilities via various exchange types.

Message scheduling (delayed delivery, TTL).

Fault tolerance through retry mechanisms and dead‑letter exchanges.

User‑friendly UI for monitoring and managing brokers.

Highly active community.

Disadvantages:

Core code written in Erlang, making source‑level customization difficult.

Throughput is lower compared to Kafka and RocketMQ due to heavier implementation.

Lacks built‑in ordering, robust persistence, replay, and has limited scalability.

04 Message Queue Selection

Kafka is ideal for scenarios requiring massive data ingestion, such as log collection for large‑scale internet services.

RocketMQ excels in financial‑grade reliability, e‑commerce order processing, and traffic‑shaping situations; it has been stress‑tested during Alibaba's Double 11 events.

RabbitMQ suits smaller‑scale projects where functional completeness and a vibrant ecosystem are more important than raw throughput.

ActiveMQ is no longer actively maintained for high‑throughput use cases and is generally not recommended.