Mastering Message Queues: Key Concepts of RabbitMQ, RocketMQ, and Kafka

RabbitMQ Components

Broker: a RabbitMQ instance.

Virtual Host (vhost): analogous to a MySQL database; a broker can host multiple isolated vhosts.

Exchange: receives messages from producers and routes them to queues.

Queue: stores messages until they are consumed.

Binding: defines the relationship between an exchange and a queue using a routing key.

Channel: a lightweight, multiplexed TCP connection used for publishing and consuming.

Connection: the TCP link between a producer/consumer and the broker.

Publisher: the message producer.

Consumer: the message consumer.

Message: consists of a header (including routing‑key, priority, etc.) and a body.

RabbitMQ Exchange Types

Direct: routes messages when the routing key exactly matches the binding key.

Fanout: broadcasts messages to all bound queues; it ignores routing keys and is the fastest exchange type.

Topic: routes messages based on pattern matching with "#" (zero or more words) and "*" (exactly one word).

TTL (Time To Live)

Specify TTL when sending a message by setting the message properties.

Specify TTL at the exchange level; messages expire after the queue’s configured timeout.

Producer Confirm Mechanism

When a broker receives a message, it sends an acknowledgment to the producer, guaranteeing reliable delivery. channel.confirmSelect() – enable confirm mode on the channel. channel.addConfirmListener(...) – add a listener to handle success or failure and optionally retry or log.

Return Listener

Used to handle undeliverable messages when the target exchange does not exist or the routing key cannot be matched. Setting the mandatory flag to true makes the broker return such messages to the producer.

Consumer ACK/NACK

Consumers can manually acknowledge successful processing; otherwise, they can reject (NACK) and decide whether to requeue the message.

public void basicNack(long deliveryTag, boolean multiple, boolean requeue)

Dead Letter Queue (DLX)

A dead‑letter queue receives messages that have been rejected, expired, or exceeded the maximum retry count, allowing further inspection or manual handling.

RocketMQ Core Concepts

Broker: stores and forwards messages.

Topic: first‑level classification of messages.

Tag: second‑level label for finer granularity.

MessageQueue: a physical queue under a topic; multiple queues enable horizontal scaling.

NameServer: lightweight service (similar to ZooKeeper) that stores routing information.

Producer: supports synchronous, asynchronous, and one‑way sending.

Consumer: supports PUSH/PULL modes and both clustering and broadcasting consumption.

Group: a set of producers or consumers sharing the same logical role.

Offset: a 64‑bit index used to locate messages within a queue.

RocketMQ High Availability

Supports master‑slave replication; masters handle reads/writes, slaves handle reads only. Consumers automatically switch to a slave when the master is unavailable. Producers can be configured with multiple master brokers for write HA.

Load Balancing

Producers round‑robin messages across all message queues, which may reside on different brokers. Consumers achieve load balancing by ensuring the total number of queues is greater than or equal to the number of consumer instances; excess consumers will not receive queues.

Kafka Core Concepts

Broker: a Kafka server node; multiple brokers form a cluster.

Topic: logical category of messages.

Partition: a sub‑segment of a topic; messages within a partition are ordered.

Producer: publishes messages to brokers.

Consumer: reads messages from brokers.

ConsumerGroup: a group of consumers where each partition is consumed by only one member.

Leader: the replica responsible for reads and writes of a partition.

Follower: replicas that sync from the leader.

Offset: the position of a consumer within a partition.

Kafka Controller and Partition Leader Election

One broker is elected as the controller via ZooKeeper; it manages broker, topic, and partition metadata. When a leader fails, the controller selects a new leader from the in‑sync replica (ISR) set, unless unclean.leader.election.enable allows election from out‑of‑ISR replicas.

Consumer Rebalance

Triggered when consumer group membership changes, partitions are added, or topics are subscribed/unsubscribed. Kafka supports three partition‑assignment strategies: range (default), round‑robin, and sticky (preserves previous assignments when possible).

Producer Message Publishing in Kafka

Messages are appended sequentially to partition logs (high‑throughput sequential writes). The routing key is hashed: hash(key) % partitionCount. The leader writes to its log, followers pull and ACK, and the high‑watermark (HW) advances only after all ISR replicas have persisted the record.