Understanding Message Queues: Execution Principles and Choosing the Right MQ

Understanding Message Queues: Execution Principles and Choosing the Right MQ

1. Background

In high‑concurrency, high‑throughput internet scenarios, Message Queues (MQ) are often used as infrastructure to handle message forwarding, traffic shaping, and asynchronous transaction processing. For operations that do not require real‑time responses, such as user registration, the request‑response flow can be off‑loaded to an MQ, reducing the visible latency for the end user.

The registration process takes about 55 ms for the request‑response path, while the subsequent message consumption and SMS sending can take over 5 seconds, which is why the heavy‑lifting is delegated to an MQ.

2. Components of a Message Middleware

Broker : the message server that runs as a service on the server side, providing core message routing for business systems.

Producer : the business initiator that creates messages and sends them to the broker.

Consumer : the business processor that retrieves messages from the broker and executes the corresponding logic.

Topic : a publish/subscribe channel where multiple producers publish messages and the MQ distributes them to subscribed consumers.

Queue : a point‑to‑point channel where a specific producer sends messages to a specific queue and a specific consumer receives them.

Message : the payload, encoded according to a communication protocol, that carries business data.

3. Communication Models

Point‑to‑Point (PTP) : messages are sent to a queue; each message is consumed by only one consumer, preventing duplicate consumption.

Publish/Subscribe (Pub/Sub) : messages are published to a topic; all subscribed consumers receive a copy of the message, enabling broadcast.

4. Advantages of Message Middleware

System decoupling: services interact via messages instead of direct calls, reducing coupling.

Peak shaving and improved response time: time‑critical tasks stay synchronous, while less urgent tasks are queued for asynchronous processing.

Ordered processing: FIFO handling ensures that long‑running tasks are processed sequentially.

Support for big‑data architectures: MQ integrates with real‑time processing pipelines to provide performance guarantees.

5. Common Protocols

AMQP, MQTT, STOMP, XMPP, and other custom TCP/IP‑based protocols.

6. Rich Open‑Source MQ Landscape

Popular open‑source options include ActiveMQ, RabbitMQ, Kafka, RocketMQ, and ZeroMQ. Selection depends on factors such as data volume, latency tolerance, dead‑letter handling, and persistence requirements.

6.1 Mainstream MQ Introduction

RabbitMQ : implemented in Erlang, follows the AMQP protocol, widely used for reliable message routing.

RocketMQ : Java‑based, high‑throughput, high‑availability solution originally from Alibaba, now an Apache project.

Kafka : Scala/Java‑based distributed log system, supports massive throughput, horizontal scaling, and is heavily used in log collection and stream processing.

6.2 Feature Comparison (summary)

Development language: Erlang (RabbitMQ), Java (RocketMQ), Scala (Kafka).

Supported protocols: AMQP (RabbitMQ), custom (RocketMQ), TCP‑based custom (Kafka).

Message storage: RabbitMQ – memory + disk (limited); RocketMQ – disk (large); Kafka – memory + disk + DB (large).

Throughput: RabbitMQ – tens of thousands QPS; RocketMQ & Kafka – >100 k QPS.

Latency: RabbitMQ – µs; RocketMQ & Kafka – ms.

Message replay: RabbitMQ – none; RocketMQ – time‑point replay; Kafka – offset/timestamp replay.

Retry: RabbitMQ – via auto‑ack false; RocketMQ – native support; Kafka – via replay.

Security: both support TLS/SSL and ACLs.

6.3 Selection Guidance

6.3.1 System Scale

For small‑to‑medium systems, RabbitMQ is recommended due to its rich feature set and ease of use. Large‑scale systems may choose RocketMQ or Kafka, both offering >100 k QPS.

6.3.2 Business Scale

Custom business requirements (e.g., Java source optimization) favor RocketMQ, while log‑centric workloads favor Kafka.

6.3.3 Functional Layer

Feature matrix highlights differences such as priority queues (supported by RabbitMQ), delay queues (both support), dead‑letter handling (both), and consumer models (push + pull for RabbitMQ, pull‑only for Kafka).

6.3.4 Performance

Kafka generally delivers higher throughput (10⁵–10⁶ QPS) than RabbitMQ (10⁴ QPS). Latency is lower for RabbitMQ (µs) compared to Kafka (ms), but most MQ use‑cases tolerate ms‑level latency.

6.3.5 High Availability

RabbitMQ achieves HA via mirrored queues; Kafka provides strong durability through replication and leader election via Zookeeper.

6.3.6 Operations Management

Monitoring tools include rabbitmq_management, Prometheus, DataDog for RabbitMQ, and Kafka Manager, Burrow, Confluent Control Center for Kafka. Disaster recovery can be handled with RabbitMQ Federation/Shovel or Kafka MirrorMaker/uReplicator.

6.3.7 Community and Ecosystem

Both Kafka and RabbitMQ have vibrant open‑source ecosystems; RocketMQ benefits from Alibaba’s backing.