Design Principles of a High‑Performance Message Broker (RocketMQ)

When business volume surges, operations such as new‑user registration that previously required only a simple SMS become a performance bottleneck, needing additional services like push notifications and coupon distribution, which increase latency and coupling.

Bill identifies three core problems: 同步 , 耦合 , and 流量暴增时系统被压垮的风险 . The synchronous flow forces the registration service to wait for downstream modules, tightly couples them, and risks system collapse under traffic spikes.

To solve this, a middle‑layer queue is introduced, turning the synchronous process into an asynchronous producer‑consumer model. The registration request is placed into a queue, allowing the service to return immediately while downstream services consume the event, achieving asynchronous processing, decoupling, and traffic‑shaping (削峰).

Using a simple JDK Queue is insufficient because it creates tight coupling, risks message loss on crashes, and cannot be shared across multiple consumers. Therefore, a dedicated broker component is designed to persist messages, provide high availability, and meet high‑throughput requirements (e.g., 100k TPS).

The broker stores messages sequentially in a commitlog file to ensure fast disk writes. To avoid costly disk I/O, the system leverages the OS page cache and mmap to map files into the process address space, eliminating extra memory copies and enabling zero‑copy reads/writes.

Because message sizes vary, a fixed‑size consumeQueue index file records each message’s commit offset , size , and tag hashcode . This 20‑byte entry allows rapid binary search to locate a consumer’s progress and retrieve the corresponding message from the commitlog.

For scalability, the broker supports multiple consume queues (sharding) and topics. A topic groups related messages, while a tag provides finer‑grained filtering within a topic. Consumers subscribe to specific topic‑tag combinations, and the broker uses the stored tag hashcode to filter messages efficiently.

High availability is achieved through master‑slave replication and, in newer versions, a Raft‑based dledger mode with multiple replicas. Producers and consumers discover brokers via a nameserver cluster, which maintains routing information (topic‑to‑broker mappings, queue counts) and handles dynamic broker registration and health monitoring.

Overall, the design combines asynchronous queuing, persistent commitlog, mmap‑backed file access, sharded consume queues, topic/tag routing, and a nameserver‑based discovery mechanism to build a high‑performance, highly available messaging system akin to RocketMQ.