Design and Implementation Differences between Pulsar, Kafka, RocketMQ, and InLong‑TubeMQ

As a representative of next‑generation message middleware, Pulsar incorporates forward‑looking design choices that address common architectural patterns, use cases, and operational challenges found in existing solutions such as Kafka, RocketMQ and Tencent’s open‑source InLong‑TubeMQ.

1. Cloud‑Native Multi‑Tenant Design

Pulsar natively supports multi‑tenant topics with a hierarchical naming scheme persistent://tenant/namespace/topic , which consists of four parts: domain (persistent or non‑persistent), tenant, namespace, and topic name. This design enables fine‑grained isolation and easier management in cloud environments, whereas Kafka, RocketMQ and InLong‑TubeMQ treat topics as flat entities and require additional operational conventions to separate users or departments.

2. Multi‑Level Flow Control

Pulsar provides flow‑control at broker, namespace and topic levels, covering production, consumption, client connections and storage quotas. Competing systems offer far fewer controls, making Pulsar more adaptable to diverse traffic patterns.

3. Compute and Storage Separation

Pulsar’s brokers are stateless; all message data is stored in a BookKeeper cluster (bookies). RocketMQ brokers store messages locally in a single commit‑log file (with optional storage‑separation in newer versions). Kafka brokers also use a write‑ahead log per partition, while InLong‑TubeMQ stores each topic in a single file on the broker.

4. Partitioning, Storage and Consumption

All four systems support partitioned topics, but their partition semantics differ. Pulsar partitions are global across the cluster and each partition can be consumed by multiple consumers within a subscription, improving parallelism. RocketMQ’s partitions (queues) are scoped to a broker‑pair, Kafka’s partitions are assigned a leader per partition, and InLong‑TubeMQ’s partitions behave like Kafka’s but store all partitions of a topic together.

For direct partition access, Pulsar uses names such as persistent://tenant/namespace/TopicA-partition-n . Consumers can subscribe to a partition or let the client router distribute messages based on keys.

Pulsar’s subscription model links each partition to all consumers in a subscription group, allowing the broker to push messages based on each consumer’s receive‑queue capacity. In contrast, Kafka, RocketMQ and InLong‑TubeMQ use pull‑based consumption (RocketMQ implements a long‑pull push hybrid).

5. Metadata Storage

Pulsar relies on ZooKeeper for metadata, while RocketMQ uses a stateless NameServer, Kafka is gradually removing its ZooKeeper dependency, and InLong‑TubeMQ uses a centralized service with limited ZooKeeper reliance. Heavy dependence on ZooKeeper can become a single point of failure in large deployments.

6. Summary

Pulsar’s cloud‑native multi‑tenant architecture, flexible flow‑control, stateless brokers, sophisticated partition‑to‑consumer mapping, and rich admin APIs make it well‑suited for large‑scale, multi‑tenant cloud scenarios, though it still inherits ZooKeeper’s availability risks. Each middleware has its own strengths; the choice should align with specific business requirements, with Pulsar being recommended for new cloud‑native projects at the author’s organization.