Why RocketMQ Solves Core Messaging Challenges – Architecture and Features Explained

1. What Problems Must a Message Middleware Solve?

Publish/Subscribe

Publish/subscribe is the most basic function of a message middleware, distinguishing it from traditional RPC communication.

Message Priority

Priorities are represented by integers; higher‑priority messages are delivered first. RocketMQ persists all messages, making strict priority sorting costly, so it does not natively support priority but allows a workaround by configuring separate high‑priority and normal queues.

Priority handling can be categorized into two approaches:

Use multiple topics to represent coarse priority levels (high, medium, low), which satisfies most use cases with some precision loss.

Strict integer‑based priority (0‑65535) requires extensive sorting and can severely impact performance; it should be used only when truly needed.

Message Order

Ordered consumption ensures that related messages (e.g., order creation → payment → completion) are processed in the exact sequence they were sent, while different orders can be processed in parallel. RocketMQ guarantees strict ordering.

Message Filter

Broker‑side filtering reduces unnecessary network traffic to consumers but adds load and complexity to the broker.

Consumer‑side filtering is fully customizable by the application but may transmit many irrelevant messages.

Message Persistence

Common persistence methods include:

Database (e.g., MySQL)

Key‑Value stores (e.g., LevelDB, Berkeley DB)

File‑based logs (e.g., Kafka, RocketMQ)

In‑memory snapshots (e.g., beanstalkd)

RocketMQ leverages the Linux file‑system cache to achieve high performance.

Message Reliability

Reliability can be affected by various failure scenarios:

Normal broker shutdown

Broker crash

OS crash

Power loss with immediate recovery

Hardware failure (CPU, motherboard, memory)

Disk failure

Scenarios (1‑4) allow RocketMQ to retain most messages (depending on sync/async flush). Scenarios (5‑6) are single‑point failures; asynchronous replication can keep ~99% of messages, while synchronous double‑write eliminates loss at the cost of performance.

Since version 3.0, RocketMQ supports synchronous double‑write.

Low‑Latency Messaging

When the broker’s buffer is not saturated, RocketMQ uses long‑polling Pull to deliver messages in real time, achieving latency comparable to Push.

At‑Least‑Once Delivery

Each message must be delivered at least once. Consumers pull messages, process them, and acknowledge only after successful handling, ensuring the guarantee.

Exactly‑Once Delivery

Prevent duplicate sending at the producer side.

Prevent duplicate consumption at the consumer side.

Achieving both in a distributed system incurs high overhead; RocketMQ relies on business‑level deduplication (idempotent consumption) and rarely experiences duplicates except under network anomalies.

What Happens When a Broker’s Buffer Is Full?

RocketMQ treats the buffer as an effectively unbounded queue backed by disk; expired data (e.g., older than three days) is periodically removed, so overflow is handled by time‑based eviction rather than rejection.

Back‑track Consumption

Consumers can re‑consume already‑processed messages by specifying a timestamp (precision to milliseconds), enabling both forward and backward replay.

Message Accumulation

Message queues absorb traffic spikes, requiring sufficient accumulation capacity. Two accumulation modes exist:

In‑memory buffer overflow, where messages may be dropped according to a policy.

Persistence to disk (DB, KV store, file logs), where I/O throughput determines post‑accumulation performance.

Evaluation criteria include total capacity, impact on throughput, effect on consumers, and disk‑read performance.

Distributed Transactions

Traditional XA/JTA rely on KV stores for two‑phase commit. RocketMQ avoids KV lookups by using the message offset as the identifier, which can increase dirty‑page pressure.

Scheduled Messages

RocketMQ supports delayed delivery at predefined levels (e.g., 5 s, 10 s, 1 min) but not arbitrary timestamps, as precise scheduling would require costly sorting and persistence.

Message Retry

When consumption fails, RocketMQ offers retry mechanisms:

Application‑level errors (e.g., deserialization failure) – skip or schedule a delayed retry.

Downstream service unavailability – pause consumption (e.g., sleep 30 s) to reduce broker load.

RocketMQ Overview

RocketMQ is a high‑performance, highly reliable, real‑time, distributed queue‑based message middleware.

Supports distributed producers, consumers, and topics.

Ensures strict message ordering.

Offers rich pull models and horizontal scaling for subscribers.

Provides real‑time subscription and massive message accumulation capabilities.

Has minimal external dependencies.

What Is RocketMQ?

Physical Deployment Structure

Key components:

NameServer: Stateless, cluster‑deployable, no state synchronization.

Broker: Master‑Slave architecture; each Master may have multiple Slaves, identified by BrokerId (0 = Master).

Producer: Stateless, connects to a random NameServer to obtain routing info and communicates with the responsible Master.

Consumer: Connects to a NameServer, retrieves routing, and can subscribe to Master or Slave based on broker configuration.

Logical Deployment Structure

Key concepts:

Producer Group: Identifies a set of producers; aids management and transaction handling.

Consumer Group: Identifies a set of consumers; enables load‑balanced or broadcast consumption.

Data Storage Structure

RocketMQ separates data and index files, reducing file, I/O, and memory consumption, which supports massive data volumes and high concurrency while maintaining low latency and strong horizontal scalability.