RocketMQ Storage HA Deep Dive: CommitLog Mechanics to Production Controller Failover

Why RocketMQ High Availability Is Fundamentally a Storage Issue

Many teams equate HA with multiple NameServers and Brokers, which only secures routing availability. True HA must answer four storage‑level questions: (1) after an ACK, is the message only in memory, on disk, or replicated? (2) When the master crashes, can a new writer be elected and can previously ACKed messages be fully read? (3) How do disk jitter, PageCache blockage, or slow replication cause throughput cliffs? (4) After recovery, how are CommitLog, ConsumeQueue and IndexFile re‑aligned to avoid logical‑queue inconsistency?

From an architect’s view, HA balances three dimensions: data consistency, service continuity, and performance cost.

A Typical Production Incident that Exposes All Weak Points

During a large promotion, broker-a experienced disk I/O jitter. The sequence was:

Producers kept writing to broker-a.

PageCache on the master accumulated, flushing threads fell behind.

Slave replication lag increased, causing the HA client to return ACKs slower.

Business threads blocked on synchronous replication, causing send‑RT spikes.

The master process was killed by the OOM killer or host failure.

NameServer removed the old route, but the traditional master‑slave model could not auto‑promote the slave.

Clients continued to use the stale route within the refresh window, leading to massive timeouts.

Manual failover restored write traffic, but messages that had not yet been fully replicated were lost.

This incident shows that a robust HA design must answer five questions about write success points, replication points, master‑switch authority, client visibility latency, and avoiding “half‑recovered” data.

RocketMQ Storage Model Overview

The storage layer consists of coordinated files: CommitLog: the physical log where all messages are appended sequentially. ConsumeQueue: a logical index per Topic/Queue that stores CommitLog Offset, Message Size and Tag HashCode. IndexFile: a key‑based search index. Checkpoint: records flush timestamps and recovery boundaries. Abort: indicates whether the broker shut down cleanly.

Only CommitLog is the source of truth; ConsumeQueue and IndexFile are derived structures that are rebuilt during recovery.

CommitLog Deep Dive: Why Sequential Writes Enable High Throughput

4.1 Sequential Append Is the Core Premise

RocketMQ achieves high write performance by enforcing four constraints:

All messages are appended only to the file tail.

No random inserts or in‑page updates.

Storage structures are fixed‑size and batch‑oriented.

Writes rely on the OS PageCache to aggregate data.

Benefits:

Disk seeks are minimal, allowing throughput near device limits.

User threads only perform memory copies, avoiding per‑message synchronous I/O.

4.2 mmap + PageCache – The Performance Source and Failure Starting Point

RocketMQ maps CommitLog files into the JVM via MappedFile. The write path is:

业务线程 → JVM 堆外/堆内编码消息 → MappedByteBuffer → OS PageCache → flush/fsync → Disk

Three often‑overlooked points:

Write success does not equal disk persistence when async flush is used; data may still reside in PageCache.

PageCache is finite; excessive dirty pages cause kernel write‑back to throttle user threads.

Disk slowdown may manifest as rising sendRT, putMessageEntireTimeMax and replication delay before any I/O saturation.

4.3 CommitLog File Rolling

CommitLog consists of fixed‑size (commonly 1 GB) files, which simplifies recovery, offset lookup and hot‑cold data lifecycle management.

4.4 Production‑Grade Write Pipeline

The eight steps a broker follows to handle a message are:

Validate Topic, size, permissions and broker state.

Select a writable MappedFile.

Encode the message header and body.

Append to CommitLog.

Depending on the flush mode, wait for disk persistence.

Depending on the replication mode, wait for slave acknowledgment.

Return the send result to the producer.

Asynchronously build ConsumeQueue and IndexFile.

Steps 1‑6 define the ACK semantics; steps 7‑8 determine when the message becomes visible to consumers.

ConsumeQueue, IndexFile and Recovery Consistency

5.1 ConsumeQueue Is a Logical Index, Not the Message Body

Each Queue has its own ConsumeQueue file storing only offset, size and tag hash. Consumers first locate the offset via ConsumeQueue and then fetch the full message from CommitLog. Thus ConsumeQueue behaves like a sparse sequential index.

5.2 ReputMessageService Controls Visibility

After a successful CommitLog write, the background thread ReputMessageService scans from the last processed position and incrementally builds ConsumeQueue and IndexFile. Consequently, a producer may receive an ACK while the consumer still cannot read the newest message—a short derived‑build window, not data loss.

5.3 Recovery Starts From CommitLog

When a broker exits abnormally, the typical inconsistency is that CommitLog has advanced but ConsumeQueue / IndexFile have not caught up. Recovery proceeds by:

Checking the Abort file to detect abnormal shutdown.

Using Checkpoint to locate a safe recovery point for CommitLog and ConsumeQueue.

Truncating dirty tail files.

Redistributing from CommitLog to rebuild ConsumeQueue and IndexFile.

CommitLog is the ultimate authority for storage recovery.

Flush Mechanisms and ACK Semantics

6.1 Async vs Sync Flush

ASYNC_FLUSH

: ACK after data reaches PageCache; risk – recent data may be lost on crash. SYNC_FLUSH: ACK after fsync; higher latency, lower TPS.

Async flush suits logging or weak‑consistency notifications; sync flush fits orders, payments and financial flows.

6.2 Sync Flush vs Sync Replication

Sync flush guarantees local disk persistence, protecting against broker or host crashes.

Sync replication guarantees that a replica copy exists, protecting against single‑node loss.

6.3 ACK Matrix

Async flush + async replication → ACK only in master memory (highest risk).

Async flush + sync replication → Master memory + some slave memory (still risky).

Sync flush + async replication → Master persisted, slave may lag (partial safety).

Sync flush + sync replication → Strongest reliability, highest performance cost.

Business teams must be told that “send success” is a business‑level guarantee, not a technical constant; it depends on broker configuration.

HA Architecture Evolution

7.1 Traditional Master‑Slave

Features: master writes, slave pulls CommitLog, NameServer only discovers routes, manual failover. Simple, compatible, moderate performance, but cannot auto‑promote and incurs high operational cost for critical workloads.

7.2 DLedger (Raft‑like)

Introduces majority‑based commit, automatic leader election and clearer consistency boundaries. Benefits: strong auto‑promotion, clearer consistency, standard replication semantics. Drawbacks: higher architectural and operational complexity, extra latency, migration cost.

7.3 Controller Mode (RocketMQ 5.x+)

Separates master‑slave role management into a dedicated Controller cluster. Broker handles storage and message I/O; Controller decides the leader based on health and sync state; NameServer publishes routes. Advantages:

Automatic, health‑aware master selection.

Unified metadata governance.

Fine‑grained control via SyncStateSet, which contains only replicas that have passed health and sync‑point checks.

Controller Details

8.1 SyncStateSet

SyncStateSet

is the set of replicas eligible for synchronization guarantees. A lagging replica may remain in the replica group but be excluded from SyncStateSet, preventing it from being promoted and avoiding data rollback.

8.2 Why Controller Suits Production

It solves three core problems:

Correctness of switch – only qualified replicas can take over writes.

Speed – decision and route propagation are fast.

Observability – which replicas are excluded and why is visible.

Source‑Level Write Path Illustration

public class ReliableMessageAppendService {
    public PutMessageResult putMessage(MessageExtBrokerInner msg) {
        long begin = System.nanoTime();
        MappedFile mappedFile = mappedFileQueue.getLastMappedFile();
        if (mappedFile == null) {
            return PutMessageResult.createFail(PutMessageStatus.CREATE_MAPPED_FILE_FAILED);
        }
        appendLock.lock();
        try {
            AppendMessageResult appendResult = mappedFile.appendMessage(msg, appendCallback);
            if (!appendResult.isOk()) {
                return PutMessageResult.createFail(PutMessageStatus.UNKNOWN_ERROR);
            }
            PutMessageResult result = new PutMessageResult(PutMessageStatus.PUT_OK, appendResult);
            // 1. Handle local persistence
            if (flushDiskType == FlushDiskType.SYNC_FLUSH) {
                GroupCommitRequest flushRequest = new GroupCommitRequest(appendResult.getWroteOffset() + appendResult.getWroteBytes());
                flushService.submit(flushRequest);
                boolean flushed = flushRequest.waitForFlush(flushTimeoutMillis);
                if (!flushed) {
                    return PutMessageResult.createFail(PutMessageStatus.FLUSH_DISK_TIMEOUT);
                }
            } else {
                flushService.wakeup();
            }
            // 2. Handle replica sync
            if (replicaMode == ReplicaMode.SYNC) {
                ReplicaAckRequest replicaAckRequest = new ReplicaAckRequest(appendResult.getWroteOffset() + appendResult.getWroteBytes());
                replicaService.submit(replicaAckRequest);
                boolean replicated = replicaAckRequest.waitForAck(replicaTimeoutMillis);
                if (!replicated) {
                    return PutMessageResult.createFail(PutMessageStatus.FLUSH_SLAVE_TIMEOUT);
                }
            }
            metrics.recordPutLatency(System.nanoTime() - begin);
            return result;
        } finally {
            appendLock.unlock();
        }
    }
}

The code shows that write latency is dominated by the sum of flush wait, replica acknowledgment, routing jitter and PageCache pressure, not merely the raw file append speed.

Production‑Grade Architecture Recommendations

10.1 Cluster Topology

Controller cluster: 3 nodes, odd count, across AZs.

NameServer: ≥3 stateless nodes, separate from brokers.

Broker replica group: at least 3 replicas per core Topic.

Separate storage‑heavy broker nodes from CPU‑intensive services.

Deploy across multiple AZs within the same region.

10.2 Disk & Network

CommitLog on local NVMe or high‑performance cloud disks, not shared with system disk.

Isolate log, data and monitoring disks.

Use dedicated high‑bandwidth intra‑cluster network for replication.

Disable swap on broker nodes to avoid long‑tail jitter.

10.3 Topic Distribution

Separate core Topics from low‑priority Topics into different clusters or broker groups.

Evaluate delay, transaction and large‑message Topics individually.

Never mix high‑accumulation Topics with ultra‑low‑latency Topics on the same hot broker.

10.4 Business Constraints

Limit single message size (commonly ≤4 KB or ≤8 KB).

Prohibit embedding large JSON, Base64 images or massive lists in the payload.

Producer must provide an idempotent key.

Consumer must implement retry, dead‑letter and compensation mechanisms.

Production Configuration Samples

11.1 Broker Configuration (excerpt)

# broker.conf
brokerClusterName=TradeCluster
brokerName=broker-a
listenPort=10911
fileReservedTime=72
deleteWhen=04
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=6000000
brokerIP1=10.20.16.31
brokerIP2=10.20.16.31

# Flush settings
flushDiskType=SYNC_FLUSH
flushCommitLogTimed=false

# Flow control
diskMaxUsedSpaceRatio=75
putMsgIndexHightWater=600000
transientStorePoolEnable=false
fastFailIfNoBufferInStorePool=false

# Thread pool sizes
sendMessageThreadPoolNums=64
pullMessageThreadPoolNums=48
ackMessageThreadPoolNums=32
queryMessageThreadPoolNums=16

# Property filter & transaction handling
enablePropertyFilter=false
rejectTransactionMessage=false

11.2 JVM Parameters

-server
-Xms16g
-Xmx16g
-Xmn8g
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:+AlwaysPreTouch
-XX:+UseStringDeduplication
-XX:InitiatingHeapOccupancyPercent=30
-XX:+ParallelRefProcEnabled
-XX:+UnlockExperimentalVMOptions

11.3 OS Tuning

# Disable swap
swapoff -a

# Increase file handles
ulimit -n 655350

# Reduce dirty page write‑back thresholds
sysctl -w vm.dirty_background_ratio=10
sysctl -w vm.dirty_ratio=15

# Lower swappiness
sysctl -w vm.swappiness=10

Producer Design and Code

12.1 Design Principles

Send timeout must be shorter than the business thread’s max wait.

Explicitly handle timeout, retry and route refresh.

Provide an idempotent key to avoid duplicate side effects.

Monitor send results and record compensation data for critical messages.

12.2 Production‑Level Send Example

public class TradeEventProducer {
    private final DefaultMQProducer producer;
    public TradeEventProducer(DefaultMQProducer producer) { this.producer = producer; }
    public SendResult sendOrderCreated(OrderCreatedEvent event) throws Exception {
        String bizKey = event.getOrderId();
        Message message = new Message("trade-order-topic", "ORDER_CREATED", bizKey, toJsonBytes(event));
        message.putUserProperty("bizKey", bizKey);
        message.putUserProperty("traceId", event.getTraceId());
        message.putUserProperty("eventType", "ORDER_CREATED");
        int maxAttempts = 3;
        Exception lastException = null;
        for (int attempt = 1; attempt <= maxAttempts; attempt++) {
            try {
                SendResult result = producer.send(message, 3000);
                if (result.getSendStatus() == SendStatus.SEND_OK) {
                    return result;
                }
                lastException = new IllegalStateException("unexpected send status: " + result.getSendStatus());
            } catch (RemotingException | MQBrokerException e) {
                lastException = e;
            } catch (MQClientException e) {
                // stale route or client state, back‑off then retry
                lastException = e;
            }
            Thread.sleep(100L * attempt);
        }
        throw new IllegalStateException("send message failed, bizKey=" + bizKey, lastException);
    }
    private byte[] toJsonBytes(OrderCreatedEvent event) {
        return Jsons.toJson(event).getBytes(StandardCharsets.UTF_8);
    }
}

Key takeaways: bounded retry count, idempotent key, and explicit exception propagation.

Consumer Design and Code

13.1 Design Principles

Business idempotency is more important than MQ deduplication.

Consume logic must be re‑entrant.

Database updates and consumption state must be coordinated via idempotent tables or state machines.

Define retry limits and dead‑letter policies.

13.2 Production Consumer Example

public class InventoryReserveConsumer implements MessageListenerConcurrently {
    private final InventoryService inventoryService;
    private final ConsumeIdempotentService idempotentService;
    public InventoryReserveConsumer(InventoryService inventoryService, ConsumeIdempotentService idempotentService) {
        this.inventoryService = inventoryService;
        this.idempotentService = idempotentService;
    }
    @Override
    public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> messages, ConsumeConcurrentlyContext context) {
        for (MessageExt message : messages) {
            String bizKey = message.getUserProperty("bizKey");
            String messageId = message.getMsgId();
            try {
                if (!idempotentService.tryStart("inventory_reserve", bizKey, messageId)) {
                    continue;
                }
                OrderCreatedEvent event = Jsons.fromJson(message.getBody(), OrderCreatedEvent.class);
                inventoryService.reserve(event.getOrderId(), event.getItems());
                idempotentService.markSuccess("inventory_reserve", bizKey, messageId);
            } catch (Exception ex) {
                idempotentService.markFailed("inventory_reserve", bizKey, messageId, ex.getMessage());
                return ConsumeConcurrentlyStatus.RECONSUME_LATER;
            }
        }
        return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
    }
}

Important points: tryStart prevents concurrent duplicate processing; markSuccess/markFailed feed compensation platforms; failures return RECONSUME_LATER instead of silently swallowing exceptions.

Fault Recovery Fundamentals

Recovery is not merely “process restarts” and “client reconnects”. Four layers must be verified:

CommitLog completeness.

ConsumeQueue catch‑up.

IndexFile usability.

Client routing updated to the new master.

Missing any layer leaves the system in a “half‑recovered” state.

Typical Failure Scenarios and Playbooks

15.1 Master Crash

Symptoms: producer send‑RT spikes, timeouts, route refresh, broker group triggers master switch.

Correct steps:

Confirm Controller completed master‑slave decision.

Verify the new master belongs to SyncStateSet and has up‑to‑date replication.

Ensure NameServer routes are refreshed.

Monitor producer success rate, not just broker process state.

Audit business tables for “order committed but message not delivered”.

Common mistakes: immediate restart of the old master and assuming recovery after client route refresh.

15.2 Disk or Log Tail Corruption

Risks: corrupted tail of CommitLog, ConsumeQueue pointing to missing offsets, partial Topic availability.

Recovery principles:

Validate against healthy replicas.

Truncate corrupted tail instead of forcing reads.

Redistribute from CommitLog to rebuild ConsumeQueue/IndexFile.

Check that max offsets of critical Topics are close to pre‑failure values.

15.3 Severe Replication Lag

Causes: slave disk bottleneck, network contention, large‑message surge, write spikes exceeding disk steady‑state.

Mitigation order:

Throttle high‑priority Topics.

Monitor replication backlog growth.

Inspect slave disk iowait, await, throughput and queue depth.

Consider removing lagging replicas from SyncStateSet temporarily.

15.4 Broker “Zombie” State

Symptoms: heartbeat intermittently present, thread pools stuck, ports still open, NameServer and client view diverge.

Observations needed: putMessageEntireTimeMax Replication delay

Flush latency

Request reject count

Business send success rate

Recovery Drill Checklist

Quarterly drills should cover:

Single broker failover.

Single controller failure.

Single NameServer outage.

Replica lag scenario.

Disk full situation.

Massive message backlog.

Stale client route cache.

Metrics to record:

Master‑switch latency.

Producer error‑rate peak.

Consumer recovery latency.

ACKed‑message loss.

Duplicate‑message surge.

Monitoring & Alerting Layers

Broker Layer

putTps

getTransferredTps

sendThreadPoolQueueSize

putMessageEntireTimeMax

dispatchBehindBytes

pageCacheLockTimeMills

Storage Layer

Disk utilization.

I/O wait.

Fsync latency percentiles.

Disk queue depth.

File deletion latency.

Replication Layer

Master‑slave replication lag.

Bytes behind.

Sync‑ack timeout count.

Master‑slave switch count.

Business Layer

Producer success rate.

Producer average RT / TP99.

Consumer backlog.

Dead‑letter queue growth.

Critical compensation task count.

Often the earliest symptom is a business‑level metric (e.g., order‑creation success vs. inventory deduction) rather than an internal MQ metric.

Capacity Planning

Key dimensions to test:

Steady‑state write TPS.

Peak burst TPS and duration.

Impact of large‑message proportion.

Acceptable business failure rate and recovery time during a master switch.

Typical mis‑judgment: sufficient raw disk bandwidth does not guarantee stability because PageCache back‑pressure, replication wait, ConsumeQueue lag and GC pauses can become bottlenecks.

Kubernetes Deployment Guidance

RocketMQ requires strong state, high I/O and stable network. Recommendations:

Use local SSD or high‑performance persistent volumes.

Leverage StatefulSet with pod anti‑affinity to avoid placing replicas on the same host.

Set terminationGracePeriodSeconds large enough for graceful shutdown and flush.

Ensure stable network identifiers (headless Service) align with broker identity management.

StatefulSet Sample (simplified)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq-broker-a
spec:
  serviceName: rocketmq-broker-a-hs
  replicas: 3
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app: rocketmq-broker-a
  template:
    metadata:
      labels:
        app: rocketmq-broker-a
    spec:
      terminationGracePeriodSeconds: 120
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - rocketmq-broker-a
            topologyKey: kubernetes.io/hostname
      containers:
      - name: broker
        image: apache/rocketmq:5.3.1
        ports:
        - containerPort: 10911
        - containerPort: 10909
        env:
        - name: JAVA_OPT_EXT
          value: "-Xms16g -Xmx16g -Xmn8g -XX:+UseG1GC -XX:+AlwaysPreTouch"
        volumeMounts:
        - name: data
          mountPath: /home/rocketmq/store
        readinessProbe:
          tcpSocket:
            port: 10911
          initialDelaySeconds: 20
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 10911
          initialDelaySeconds: 60
          periodSeconds: 20
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 500Gi
      storageClassName: local-ssd

Key points: odd‑numbered controller, cross‑AZ spread, large termination grace, and dedicated storage.

Migration Path from Traditional Master‑Slave to Controller

Establish baseline metrics: core Topics, reliability requirements, current RT, backlog and recovery time.

Run gray‑scale tests on low‑risk Topics using a new replica group under Controller governance; simulate master failure and route refresh.

Validate idempotent logic and compensation pipelines.

Gradually migrate core Topics with synchronous flush and strong replication, avoiding peak business periods.

Maintain dual observation (old and new paths) until consistency is proven.

Strategy Selection for Different Business Types

Log/Telemetry: async flush, async replication – prioritize throughput and cost.

Order/Inventory/Coupon: at least sync flush, preferably sync replication – strong durability.

Payment/Funds: highest reliability – sync flush + sync replication, Controller mode, full auditability.

Five Questions Every Architect Must Answer

In the worst case, can an ACKed message be lost?

How long does the system need to resume writes after a master switch?

Do clients implement retry, idempotency and compensation?

Can the system identify half‑healthy replicas and avoid promoting them?

Has a realistic failure‑drill been performed and a recovery playbook documented?

If any answer is unclear, the system is not truly production‑ready.

Conclusion

RocketMQ storage HA is not a single configuration switch. It comprises:

CommitLog as the sole source of truth.

Flush and replication policies tightly matched to business criticality.

Clear separation of control plane (Controller) and data plane (Broker).

End‑to‑end producer/consumer reliability design.

Comprehensive fault‑drill, monitoring and recovery procedures.

Only when these three layers—storage consistency, operational governance and business‑level reliability—are jointly satisfied does RocketMQ become a production‑grade backbone rather than just a fast message queue.