Message Queue (MQ) Overview, Selection Criteria, and Comparison of Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ

Message queues are essential middleware components in distributed systems, providing asynchronous communication, traffic shaping, decoupling, caching, and high availability.

The article presents a comprehensive overview of major MQ products—Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ—including their system architectures, core terminology, and operational characteristics.

Architecture Overview

Kafka : Consists of multiple Brokers and a ZooKeeper cluster. Topics are divided into partitions, each replicated across Brokers for fault tolerance.

Pulsar : Uses a stateless Broker layer for compute and Apache BookKeeper for storage. It also relies on ZooKeeper for metadata management.

RocketMQ : Built on NameServer, Broker, Producer, and Consumer components. Supports master‑slave replication.

RabbitMQ : Implements the AMQP protocol with Exchanges, Queues, and Bindings. Supports plugins for delayed messages.

NSQ : Composed of nsqd, nsqlookupd, and nsqadmin processes, providing lightweight, decentralized messaging.

Selection Criteria

Message ordering

Scalability (horizontal/vertical)

Message retention and TTL

Fault tolerance and reliability

Latency and throughput

Routing capabilities

Support for delayed, priority, and transactional messages

Functional Analysis

Key features compared include push vs. pull consumption models, delayed queues, dead‑letter queues, priority queues, message tracing, and consumption modes (exclusive, shared, failover, key‑shared, broadcast, clustering).

Performance

Benchmark studies show Kafka generally leads in raw throughput, while Pulsar can match Kafka after tuning and offers lower latency in certain scenarios. RabbitMQ provides microsecond‑level latency but lower throughput. RocketMQ and NSQ have distinct performance trade‑offs based on architecture.

Operations & Maintenance

High availability is achieved via replication (Kafka, Pulsar, RocketMQ) or master‑slave (RabbitMQ). Cross‑region disaster recovery is natively supported by Pulsar. Scaling strategies differ: Kafka requires partition rebalancing; Pulsar benefits from storage‑compute separation; RocketMQ adds new brokers without data rebalancing; RabbitMQ and NSQ scale by adding nodes.

Common Issues & Use Cases

Typical use cases include log collection (Kafka), financial transaction processing (RocketMQ), real‑time messaging (RabbitMQ), and lightweight Go‑based services (NSQ). The article also discusses troubleshooting message backlog, ensuring exactly‑once processing, and handling delayed or transactional messages.

Conclusion

For high‑throughput, log‑oriented workloads, Kafka remains a solid choice. For scenarios demanding strong reliability, multi‑tenant isolation, or advanced features like delayed queues, Pulsar is preferable. RabbitMQ suits simple queueing with low latency, while RocketMQ excels in financial domains. NSQ offers a Go‑native, extensible option for custom needs.

ConnectionFactory factory = new ConnectionFactory(); Connection connection = factory.newConnection(); Channel channel = connection.createChannel(); channel.txSelect(); // start transaction channel.basicPublish("directTransactionExchange", "transactionRoutingKey", null, message.getBytes("utf-8")); channel.txCommit(); // commit transaction