Message Queue Evolution, Architecture, and Storage Analysis

This article outlines the historical development of message queues, beginning with the first stage (2003‑2010) focused on decoupling systems using ActiveMQ and RabbitMQ, followed by the second stage (2010‑2012) emphasizing throughput and consistency with the emergence of Kafka, and the third stage (post‑2012) where platformization led to the creation of RocketMQ and Pulsar.

It then introduces the fundamental concepts of message queues—topics, producers, consumers, and partitions—using a cafeteria analogy to illustrate how messages are produced, queued, and consumed.

The piece provides a comparative analysis of the storage architectures of three mainstream message queues: Kafka, RocketMQ, and Pulsar. For Kafka, it describes the partition‑based design, sequential disk writes, and the role of Zookeeper for metadata management.

For RocketMQ, it details the dual‑master architecture, the replacement of Zookeeper with lightweight NameServer, and the use of CommitLog, ConsumeQueue, and IndexFile for efficient storage and retrieval.

Pulsar’s design is explained as a layered and sharded architecture separating stateless brokers from storage nodes (BookKeeper), enabling high availability, dynamic scaling, and fault‑tolerant data replication.

The article also discusses performance trade‑offs, such as Kafka’s sensitivity to large numbers of topics and the impact on sequential write efficiency, and highlights how RocketMQ and Pulsar address these challenges through different storage and metadata strategies.

In conclusion, the author emphasizes that each message queue design balances trade‑offs between throughput, consistency, scalability, and operational complexity, and that selecting the appropriate system depends on specific application scenarios.