Message Queue Middleware Selection Guide: Kafka vs Pulsar vs RocketMQ vs RabbitMQ vs NSQ

Message queues are crucial middleware in distributed systems, playing important roles in high-performance, high-availability, and low-coupling system architectures. This article provides a comprehensive comparison of five popular message queue components: Kafka, Pulsar, RocketMQ, RabbitMQ, and NSQ.

Overview: Distributed systems can leverage message queues to achieve decoupling (separating upstream and downstream processes), broadcasting (one message to multiple consumers), buffering (handling traffic spikes), asynchronous processing, and message redundancy for failure recovery.

Architecture Introduction:

Kafka: Consists of multiple Brokers and a ZooKeeper cluster. Topics are divided into Partitions stored across different Brokers with replicas for reliability.

Pulsar: Has three components - stateless Brokers, stateful BookKeeper (for storage), and ZooKeeper (for metadata). Implements storage-compute separation architecture.

RocketMQ: Includes NameServer, Broker, Producer, and Consumer components. NameServer manages brokers and routing information.

RabbitMQ: Implements AMQP protocol with Exchange and Queue components connected via RoutingKey.

NSQ: Composed of nsqlookupd (for topology and discovery) and nsqd (for message receiving, queuing, and delivery).

Key Selection Criteria:

Consumption Models: Kafka and RocketMQ use Pull mode; RabbitMQ, Pulsar, and NSQ use Push mode.

Delay Queues: Pulsar supports second-level delays; RocketMQ supports fixed levels; RabbitMQ requires plugin; NSQ supports up to 2 hours; Kafka does not support.

Dead Letter Queues: Pulsar, RocketMQ, and RabbitMQ support; Kafka and NSQ do not.

Message Persistence: Kafka, RocketMQ, and Pulsar use disk storage; RabbitMQ primarily uses memory with disk fallback; NSQ uses local disk.

Message Ordering: Kafka guarantees partition-level ordering; Pulsar supports both exclusive (ordered) and shared (unordered) modes; RocketMQ requires locks; RabbitMQ and NSQ have limited ordering guarantees.

Multi-tenant: Only Pulsar natively supports multi-tenant isolation.

Performance: Kafka generally performs best for high-throughput scenarios; RabbitMQ has microsecond-level latency; Pulsar can match Kafka with proper tuning.

Conclusions: For log processing and big data scenarios requiring high throughput, Kafka remains a good choice. For business transaction data with requirements for delayed messages, queue mode consumption, and cross-region disaster recovery, TDMQ/Pulsar is recommended. For Golang-based stacks, NSQ offers customization opportunities.