Evolution and Architecture of Message Queues: Kafka, RocketMQ, and Pulsar

This article, authored by a member of Tencent's TDMQ founding team, presents a comprehensive overview of message queue (MQ) technologies spanning the past two decades. It explains the historical background, core problems each MQ solved, and the underlying architectural concepts, helping readers understand both legacy and emerging MQ designs.

1. Development Stages of Message Queues

1.1 Decoupling Stage (2003‑2010) : Early MQs such as ActiveMQ and RabbitMQ focused on breaking tight coupling between services and enabling asynchronous communication.

1.2 Throughput & Consistency Stage (2010‑2012) : With the rise of big‑data workloads, Kafka emerged to provide high‑throughput log collection and reliable delivery.

1.3 Platformization Stage (2012‑present) : Cloud and containerization trends led to the creation of Pulsar, which separates compute and storage layers for better scalability.

2. Core Concepts and Generic Architecture

The article introduces topics, producers, consumers, and partitions using a cafeteria analogy to illustrate how messages flow from production (choosing a dish) to consumption (taking the dish). It emphasizes that partitions enable horizontal scaling and are the basis for Kafka’s high throughput.

3. Storage Analysis of Mainstream MQs

3.1 Kafka : Uses a leader‑follower model per partition, with sequential disk writes to maximize PageCache utilization. The architecture diagram shows a single producer, single consumer, two partitions, and three replicas across three broker nodes.

3.2 RocketMQ : Improves on Kafka by replacing ZooKeeper with a lightweight Namesrv service and adopting a multi‑master, multi‑slave broker topology. It stores all topics in a single commit‑log file, uses ConsumeQueue for offset indexing, and IndexFile for key‑based lookups.

3.3 Pulsar : Introduces a layered architecture separating stateless brokers from the BookKeeper storage layer. It employs fine‑grained segments (shards) instead of coarse partitions, enabling better resource utilization, dynamic scaling, and seamless fault‑tolerance.

4. Comparative Insights

The article discusses trade‑offs such as Kafka’s performance degradation with a large number of topics due to loss of sequential write benefits, RocketMQ’s inability to delete individual topics because of its shared commit‑log, and Pulsar’s advantages in containerization and elastic scaling.

5. Summary

Message queue technology has continuously evolved to address decoupling, throughput, and platformization challenges. Selecting the appropriate MQ depends on specific use‑cases, performance requirements, and operational constraints. The article concludes by encouraging readers to share their experiences and offers a promotional giveaway for participation.