Overview of RabbitMQ, RocketMQ, and Kafka Components and Mechanisms

RabbitMQ

Components and Functions

Broker – a RabbitMQ instance.

Virtual Host (vhost) – analogous to a MySQL database; isolates resources within a broker.

Exchange – receives messages from producers and routes them to queues.

Queue – stores messages until they are consumed.

Binding – defines the relationship between an exchange and a queue using a routing key.

Channel – a lightweight, multiplexed TCP connection used for publishing, subscribing, and receiving messages.

Connection – the TCP link between producer/consumer and the broker.

Publisher – message producer.

Consumer – message consumer.

Message – consists of a header (including routing‑key, priority, etc.) and a body.

Exchange Types

Direct – routes messages when the routing key exactly matches the binding key.

Fanout – broadcasts messages to all bound queues; fastest among the three.

Topic – uses pattern matching with "#" (zero or more words) and "*" (exactly one word) wildcards.

TTL (Time To Live)

Set at message publish time via message properties.

Set at exchange creation; messages expire after the queue’s configured timeout.

Producer Confirmation Mechanism

Broker acknowledges receipt of a message, confirming successful delivery.

Producer listens for the acknowledgment to guarantee reliable delivery.

How to implement Confirm:

Enable confirm mode on a channel: channel.confirmSelect() Add a confirm listener via addConfirmListener to handle success or failure and optionally retry or log.

Return Listener

Handles undeliverable messages when the exchange does not exist or the routing key cannot be resolved. Setting mandatory=true forces the broker to return such messages to the producer.

Consumer ACK/NACK

Consumers should manually ACK successful processing; in case of severe failures (e.g., server crash) a manual ACK ensures reliability.

Dead‑Letter Queue (DLX)

Messages that become dead letters are automatically republished to a designated DLX, which is a normal exchange with its own queues.

RocketMQ

Alibaba’s official messaging product for high‑availability, high‑reliability scenarios.

Core Concepts

Broker – stores and forwards messages; built on Netty.

Topic – first‑level classification of messages.

Tag – second‑level classification, similar to sub‑topics.

MessageQueue – multiple queues under a topic for horizontal scaling.

NameServer – lightweight service (no inter‑communication) that registers topics and routes.

Producer – supports synchronous, asynchronous, and one‑way sending.

Consumer – supports PUSH/PULL, cluster consumption, and broadcast consumption.

Group – producer or consumer group for logical partitioning.

Offset – 64‑bit index representing a message’s position in a queue.

Delay Messages

RocketMQ supports predefined delay levels (e.g., 1s, 5s, 10s, 30s, 1m, 2m, … up to several hours).

Ordered Messages

Guarantees FIFO ordering either per partition or globally.

Transaction Messages

Provides XA‑like distributed transaction support with half‑message, commit, rollback, and compensation flows.

High‑Availability Mechanism

Master‑Slave brokers: masters handle reads/writes; slaves handle reads only.

Automatic consumer failover to slaves when a master is unavailable.

Multiple master brokers per topic for write HA.

Replication can be synchronous (strong consistency) or asynchronous (higher throughput).

Load Balancing

Producer uses round‑robin across all message queues.

Consumer instances are assigned queues; excess consumers receive no queues.

Cluster mode distributes queues evenly among consumers.

Dead‑Letter Queue

When a message exceeds the maximum retry count, it is moved to a special DLQ named %DLQ%+ConsumerGroup.

Kafka

Distributed, partitioned, replicated messaging system built on ZooKeeper.

Core Concepts

Broker – a Kafka node; multiple brokers form a cluster.

Topic – logical grouping of messages.

Partition – ordered sequence of messages within a topic.

Producer – sends messages to brokers.

Consumer – reads messages from brokers.

ConsumerGroup – ensures each partition is consumed by only one consumer in the group.

Leader – the replica responsible for reads/writes of a partition.

Follower – replicas that sync from the leader.

Offset – position of a message within a partition.

Controller

One broker is elected as the controller via ZooKeeper; it manages broker, topic, and partition metadata.

Partition Leader Election

When a leader fails, the controller selects a new leader from the ISR (in‑sync replicas) list, respecting the unclean.leader.election.enable setting.

Consumer Offset Management

Consumers periodically commit offsets to the internal __consumer_offsets topic; the topic is sharded (default 50 partitions) to handle high concurrency.

Consumer Rebalance

Triggered when consumer group membership changes, partitions are added, or topics are subscribed/unsubscribed. Strategies include range (default), round‑robin, and sticky.

Producer Publishing Flow

Discover the leader for the target partition via ZooKeeper.

Send the message to the leader.

Leader appends the message to its log.

Followers pull the message, write to their logs, and ACK the leader.

Leader updates the high‑watermark (HW) after all ISR replicas ACK and replies to the producer.

Log Segmentation

Each partition’s data is stored in segment files (default max 1 GB). Index files map offsets to log positions, and time‑index files enable timestamp‑based lookups.

Reliability Settings

Set acks=all, replication.factor, min.insync.replicas, and high retries to avoid data loss.

Disable automatic offset commits and manually commit after processing to ensure at‑least‑once delivery.

Idempotence

Message duplication can occur during send retries, broker failover, or consumer rebalance. Applications must implement idempotent processing.

Message Backlog Solutions

Introduce a dispatcher consumer that spreads backlog across multiple queues.

During low‑traffic windows, re‑import expired messages.

If disk fills, aggressively consume or discard messages to free space.

Promotional Section (Non‑Academic)

Information about community groups, QR codes, and disclaimer notices is included but does not affect the technical content.