Message Queue Architecture Comparison: NSQ, Kafka, and RocketMQ in Distributed Systems

Message queues are critical middleware in distributed systems, playing essential roles in achieving high performance, high availability, scalability, and eventual consistency. This article provides a comprehensive comparison of three popular message queue implementations: NSQ (YouZan branch), Kafka, and RocketMQ.

1. Problems Solved by MQ in Distributed Scenarios

1.1 System Decoupling

In distributed and microservices architectures, system dependencies become increasingly complex as business functions expand. Message queues can decouple services by having upstream services send data to MQ while downstream services consume messages based on their own business needs and processing capabilities.

1.2 Asynchronous Processing

Synchronous system calls face various bottleneck limitations regarding throughput, concurrency, and response time. Message queues enable asynchronous processing of non-critical or non-real-time requests, allowing message splitting during processing to improve resource utilization.

1.3 Peak Shaving

During traffic spikes, message queues can accumulate large volumes of messages, which consumers continue processing after the peak subsides. This significantly improves system throughput and stability, particularly suitable for scenarios with severe traffic fluctuations like e-commerce promotions.

1.4 Data Distribution

In distributed scenarios, a single event message may be of interest to multiple downstream systems. Most message middleware supports one-to-many consumption or message broadcasting patterns, enabling message routing based on user-defined rules.

2. Architecture Comparison of NSQ, Kafka, and RocketMQ

2.1 Multi-Node Collaboration Models

NSQ (YouZan Branch)

Original NSQ uses nsqlookup for service discovery. YouZan's branch upgraded the service awareness model by introducing ETCD for cluster management and metadata storage. nsqd and nsqlookup services register with ETCD upon startup. nsqd reports node load information to nsqlookup, which automatically performs data balancing based on load conditions.

Kafka

Kafka uses ZooKeeper for management and coordination, leveraging ZK's ordered nodes, ephemeral nodes, and watch mechanisms for load balancing, cluster management, and leader election. Each broker registers in ZK's /brokers/ids directory, and consumer group coordination requires registering consumer-partition relationships in ZK.

RocketMQ

RocketMQ uses lightweight NameServer for coordination and governance. Multiple NameServer nodes deploy independently. Each broker establishes long-lived connections to all configured NameServers, sending heartbeats every 30 seconds. NameServer checks broker存活 every 10 seconds.

2.2 Message Storage Models and Data Synchronization

NSQ (YouZan Branch)

Storage Model Optimization: Original NSQ stores messages in memory first, flushing to disk when memory accumulates. YouZan's NSQ optimizes to real-time disk persistence with data replication for reliability. Topics are partitioned with independent leader nodes for improved read/write scalability.

Consumption Model Optimization: After transformation, channels no longer store message data; they only maintain offset information for synchronized and consumed data. All channels reference the same topic disk data, eliminating data replication operations.

Consistency and Reliability: Each topic's data replica metadata is written to ETCD, which elects leader nodes. The leader synchronizes data to follower nodes. When a leader fails, ETCD triggers a watch event to select a new leader for quick failover.

Kafka

Topic and Partition Storage: Topics are logical concepts representing message collections. Each topic can have multiple partitions across different brokers. Messages are assigned an offset within their partition for ordering.

Partition Storage: Partitions are stored as files in the filesystem. Each partition directory contains a log file and two index files (offset and time indices). Log files split into segments (default 1GB).

Consumption Model: Multiple partitions enable parallel writes and concurrent consumption by consumer groups. Each partition is consumed by only one consumer within a group. Kafka stores consumer offsets in a special topic named __consumer_offsets.

Consistency and Reliability: Kafka elects a controller node via ZK ephemeral nodes to manage broker changes, topic changes, and partition management. Partition replication ensures reliability with leader handling reads/writes and followers pulling data for synchronization.

RocketMQ

Storage Design: Unlike Kafka and NSQ, RocketMQ does not use partitions or store by topic. All topic messages are written to a single file (commit log), ensuring absolute sequential I/O for maximum write performance.

Storage Files: Commit log files are split (default 1GB). Consumer queue files record offsets for each consumer group. Index files provide hash indices for message retrieval by specific properties.

Consistency and Reliability: Brokers with the same cluster name automatically form master-slave relationships. Slave servers pull unsynchronized messages from master every 5 seconds. Version 4.5.0 integrates Dledger technology (Raft-based Commit log management) for automatic failover.

3. Feature Summary and Selection Analysis

Selection Recommendations:

Log and Big Data Scenarios: Kafka is recommended as it's proven and mature in these areas.

High Topic Count: RocketMQ supports thousands of topics while Kafka and NSQ experience performance degradation after hundreds of topics. RocketMQ's mixed storage model avoids the file quantity explosion problem.

Customization Needs: NSQ is most friendly for extensions and secondary development due to its lightweight nature.

Complexity: Kafka requires ZooKeeper without a built-in management console. NSQ and RocketMQ both provide management dashboards and easier deployment.