Mastering Message Queues: Core Concepts of RabbitMQ, RocketMQ, and Kafka

RabbitMQ

RabbitMQ components: Broker, Virtual Host (vhost), Exchange, Queue, Binding, Channel, Connection, Publisher, Consumer, Message (with routing‑key, priority, etc.).

Exchange types

Direct : routing key must match exactly.

Fanout : broadcasts to all bound queues, no routing key.

Topic : pattern matching with “#” (zero or more words) and “*” (single word).

TTL

Two ways to set message expiration: via message properties when publishing, or by configuring queue TTL at creation.

Producer confirm mechanism

Enable confirm mode on a channel (channel.confirmSelect()) and add a ConfirmListener to handle ack/nack and possible retransmission.

Return listener

Handles unroutable messages when the mandatory flag is true; use channel.addReturnListener(...).

Consumer ACK/NACK

public void basicNack(long deliveryTag, boolean multiple, boolean requeue)

Manual ACK ensures reliable consumption; requeue should usually be false to avoid endless loops.

Dead‑letter queue (DLX)

Messages that exceed max retry count are routed to a DLX named %DLQ%<ConsumerGroup>.

RocketMQ

Core concepts: Broker, Topic, Tag, MessageQueue, NameServer, Group, Offset, Producer, Consumer.

Broker : stores and forwards messages; registers topics with NameServer.

Topic : logical message category; can have many MessageQueues.

Tag : secondary label for finer filtering.

MessageQueue : physical queue under a topic, enables horizontal scaling.

NameServer : lightweight metadata service (no ZooKeeper).

Producer : supports synchronous, asynchronous, and one‑way sending.

Consumer : supports push/pull, clustering, and broadcasting.

Group : producer or consumer group.

Offset : 64‑bit index of a message in a queue.

Delay messages

RocketMQ supports predefined delay levels (1s, 5s, 10s … up to 2h).

Ordered messages

Strict FIFO can be achieved with partitioned or global ordering.

Transactional messages

Three‑step flow: send half message → broker stores → local transaction → commit/rollback based on transaction status.

High‑availability

Master‑slave brokers; consumers automatically switch to a slave if the master fails. Producer writes to multiple masters for HA.

Load balancing

Producer rounds‑robin across MessageQueues; consumer instances share queues evenly.

Dead‑letter queue

Failed messages after max retries are sent to a DLQ specific to the consumer group.

Kafka

Distributed, partitioned, replicated messaging system built on ZooKeeper.

Broker : a Kafka node; a cluster consists of multiple brokers.

Topic : logical category; each topic is split into partitions.

Partition : ordered log; each has a leader and followers.

Producer : sends messages to the leader of a partition.

Consumer and ConsumerGroup : each group consumes a partition once.

Offset : position of a consumer in a partition, stored in __consumer_offsets.

Controller : broker elected via ZooKeeper to manage partition leadership.

Leader election & replication

Followers replicate the leader; ISR list determines which replicas are in‑sync. Configurations such as replication.factor, min.insync.replicas, acks=all, and retries=MAX ensure durability.

Consumer rebalance

Triggered by changes in group membership, partition count, or subscribed topics; strategies include range, round‑robin, and sticky.

Producer reliability

acks=all guarantees that a message is written to all in‑sync replicas before the producer receives an ack.

Consumer offset handling

Disable auto‑commit and manually commit after processing to avoid data loss; duplicate consumption may occur and should be handled idempotently.

Common challenges & solutions

Ensuring ordered consumption: single consumer per queue (RabbitMQ) or hash‑based partitioning (RocketMQ, Kafka).

Implementing delayed consumption: TTL/dead‑letter (RabbitMQ), built‑in delay levels (RocketMQ), or separate delay topics (Kafka).

Guaranteeing reliable delivery: confirm/ack mechanisms, persistent messages, and appropriate replication settings.

Idempotency: detect duplicate Message‑ID and apply idempotent processing.

Message backlog: redistribute load across more queues or consumers; use dead‑letter queues for expired messages.