How RocketMQ Achieves High‑Availability Storage and Fast Fault Recovery

Introduction

RocketMQ stores messages in a fixed‑length, append‑only CommitLog and builds a lightweight index called ConsumeQueue . This design enables fast recovery after a broker crash or restart while guaranteeing data consistency.

1. Fault Recovery – Broker Restart

Recovery workflow

File detection and validation

Traverse ${ROCKET_HOME}/store to locate CommitLog, ConsumeQueue and IndexFile files.

Validate each file by checking its magic number and physical length.

Recover CommitLog and ConsumeQueue

Identify the last complete CommitLog file or the previous checkpoint.

Sequentially scan messages from that offset, parsing Topic, QueueId, physical offset, length, tag hash, etc.

Rebuild the 20‑byte ConsumeQueue entries and append them to the corresponding Topic’s ConsumeQueue file.

Performance optimizations

Use the ${ROCKET_HOME}/store/checkpoint file to record the last flush point.

The broker maintains a RecoverPoint indicating the latest recoverable offset, so only data after the checkpoint is scanned.

Parallelize CommitLog scanning with multiple threads to rebuild ConsumeQueue and IndexFile, accelerating TB‑scale recovery.

IndexFile recovery

IndexFile is rebuilt by scanning the CommitLog similar to ConsumeQueue. It supports queryMessage queries and has lower recovery priority than ConsumeQueue.

Partial write handling

If a crash occurs while writing a CommitLog entry, the recovery logic discards the half‑written message by verifying the recorded length and CRC, preserving file integrity and order.

Flush strategy trade‑offs

Asynchronous flush : high throughput, but a crash may lose a small number of unflushed messages.

Synchronous flush : guarantees no data loss, at the cost of higher latency and reduced throughput.

2. High‑Availability (HA) – Master‑Slave Replication

Replication modes

Synchronous replication : the master writes a message and waits for at least one slave to persist and ACK before returning SEND_OK. Provides strong consistency and no message loss, but adds latency because the write speed depends on the slave’s network and disk.

Asynchronous replication : the master writes and returns SEND_OK immediately; replication to slaves happens later. Offers high performance and low latency, but messages not yet synced may be lost if the master crashes.

Configuration example

brokerRole=ASYNC_MASTER   # asynchronous master
brokerRole=SYNC_MASTER    # synchronous master
brokerRole=SLAVE          # slave

HA workflow and read/write separation

Data synchronization : when a slave starts, it reports the maximum physical offset it has replicated. The master then pushes CommitLog data from that offset onward, keeping the slave up‑to‑date.

Read/write separation : by default, the master handles both write and consume requests. Consumers can pull messages from slaves to achieve read load balancing, though slaves may lag behind the master. Lag thresholds can be configured to balance latency and consistency requirements.

Failover

Traditional mode : manual promotion of a slave to master via operators or scripts.

DLedger mode : based on the Raft protocol, nodes automatically elect a new master after the old one crashes. The new master is announced to NameServer and clients, achieving minute‑level automatic failover.

Network partition handling

DLedger avoids split‑brain by requiring a majority vote before electing a master, ensuring a unique master.

Traditional master‑slave setups need external coordination or manual intervention to resolve partitions.

3. Operational Best Practices

Rolling restart / upgrade : restart brokers one by one to avoid full downtime; ensure at least one replica remains online in master‑slave mode.

Key metrics to monitor

Sync replication lag – measures delay between master and slave.

Flush latency – time taken for messages to be persisted to disk.

RecoverPoint difference – amount of data that must be rebuilt after a restart.

Backup strategy : perform regular incremental backups of the CommitLog to protect against simultaneous node failures or operational errors.

4. Summary of Trade‑offs

Fault recovery goal : guarantee consistency of a single broker’s storage files after a crash.

HA goal : keep the overall service available when a host fails, eliminating a single point of failure.

Core techniques

Fault recovery – fixed‑length files, append‑only writes, sequential scanning, checkpoint files, multi‑threaded index rebuilding.

HA – master‑slave data sync (sync or async), failover (manual or DLedger automatic), read/write separation.

Performance impact

Recovery time grows linearly with the amount of ConsumeQueue data to rebuild; asynchronous flush has minimal runtime impact.

Synchronous replication adds noticeable write latency, while asynchronous replication’s impact is negligible.

Data consistency

Node‑internal consistency is ensured after recovery.

Synchronous replication provides strong consistency; asynchronous replication may lose messages not yet synced.

Design trade‑offs

Recovery speed vs. runtime performance (flush strategy).

Data consistency vs. write performance (replication strategy).

Conclusion

RocketMQ’s storage HA is a configurable, multi‑layer system. By selecting the appropriate flush mode (asynchronous or synchronous) and replication mode (asynchronous, synchronous, or DLedger), developers can balance throughput and reliability to meet the requirements of different business scenarios.