Understanding Message Queues: Concepts, Models, and Storage in MQ, Kafka, and RocketMQ

Message Queue (MQ)

A message queue (MQ) is middleware that receives, stores, and forwards messages, commonly used to achieve asynchronous processing, traffic shaping, and system decoupling.

Key Scenarios

Asynchronous: Service A sends a message to Service B without waiting for an immediate response.

Traffic Shaping: High‑traffic services (e.g., flash sales) place requests into a queue, which are processed at a manageable rate.

Decoupling: Producers and consumers interact through the queue rather than directly.

In essence, MQ stores and forwards messages, and its usage depends on the problems it solves.

MQ Conceptual Model

MQ promises three capabilities: receive messages, store them, and forward them.

Receiving Messages

When sending a message, the sender must specify a destination, similar to a phone number. In many business scenarios the sender does not know the exact consumer; instead, the message is sent to a middle layer (the MQ) using a topic . Consumers later request messages from that topic.

Consuming Messages

Consumers are defined as services that pull messages from the MQ. They indicate which topic (or partition) they want to consume, allowing the MQ to route the appropriate messages.

The final model shows producers and consumers interacting via MQ topics, achieving decoupling.

Storage

Efficiency, not capacity, is the primary concern for storage. High‑performance storage typically converts random writes into sequential writes and performs batch flushing (集中刷盘).

Why Convert Random Writes to Sequential Writes?

Mechanical hard drives incur significant seek time; sequential writes require only one seek, greatly improving throughput.

Why Batch Flush?

Each flush triggers a system call; batching writes reduces overhead, and both HDDs and SSDs benefit from writing larger blocks.

Kafka

Kafka introduced high‑performance messaging with concepts such as consumer groups and partitions. A partition corresponds to a single file that is written sequentially and flushed periodically, enabling efficient storage.

Consumer Groups

Consumer groups partition the workload among multiple instances of a service, preventing competition for the same messages.

Partitions

Splitting a topic into many partitions allows each instance to consume from its own partition, avoiding contention.

Producer Groups

Producers specify a target partition (or let the system assign one) when sending messages, ensuring messages are placed correctly within the topic.

RocketMQ

RocketMQ borrows Kafka’s storage design but replaces partitions with ConsumeQueue . The ConsumeQueue stores only the offset, size, and tag of each message, while the actual message bodies reside in a single CommitLog file.

Thus, RocketMQ writes all message bodies sequentially to one file, while many small index files (ConsumeQueues) point to locations within that file.

Storage Comparison

Message Body Storage

Kafka creates one file per partition, leading to many files and potential random writes as the number of partitions grows. RocketMQ writes all bodies to a single CommitLog, preserving sequential writes.

Impact of Index Files

ConsumeQueue entries are fixed‑size (20 bytes) and are written to the page cache. Because they are small, they occupy less cache and are flushed less frequently, avoiding the random‑write penalty that many large Kafka partitions suffer.

Key Takeaways

ConsumeQueue’s fixed‑size entries result in low cache pressure and infrequent flushing.

Kafka’s many partitions can cause random writes during concurrent flushes, whereas RocketMQ’s single CommitLog maintains sequential writes.

Thank you for reading; hope this helps :) Source: blog.csdn.net/loulanyue_/article/details/89424013