RocketMQ High Availability Mechanism: DLedger and Raft-Based Leader Election

This article provides a comprehensive analysis of RocketMQ's high availability mechanism, with a particular focus on DLedger's implementation of Raft-based leader election. The content begins by establishing the importance of high availability in distributed message middleware, discussing key requirements such as protocol stability, data persistence, message distribution, high availability, and high reliability.

The article then examines common cluster implementation approaches in message middleware, including master-slave replication (asymmetric) and peer-to-peer network (symmetric) architectures. It explains how traditional master-slave setups work with one master node handling writes while multiple slave nodes handle reads, and how peer-to-peer systems distribute responsibilities across all nodes without a single point of failure.

The core discussion centers on RocketMQ's high availability evolution. Prior to version 4.5, RocketMQ lacked automatic node promotion, meaning if the master node failed, the cluster would lose write capability. Version 4.5 introduced DLedger mode with automatic leader election. The article explains that DLedger fundamentally implements the Raft consensus algorithm.

The technical implementation details are thoroughly covered, starting with CommitLog - RocketMQ's core data storage component that provides sequential write capabilities for message persistence. The article then contrasts this with DLedgerCommitLog, which extends CommitLog by adding distributed log storage capabilities based on Apache DistributedLog (DLedger).

The broker startup process is explained, showing how DLedgerRoleChangeHandler is registered as a leader election callback when DLedger is enabled. The article includes detailed code snippets showing the initialization process, including the creation of DefaultMessageStore, DLedgerRoleChangeHandler, BrokerStats, and MessageStorePluginContext objects.

The DLedgerServer component is introduced, which uses openmessaging's DLedgerServer to handle leader election operations. The startup sequence is detailed, including the initialization of dLedgerStore, fsmCaller (state machine), dLedgerRpcService, dLedgerEntryPusher, and dLedgerLeaderElector.

The Raft algorithm implementation is explained through the three roles: Leader, Follower, and Candidate. The Leader's responsibilities include sending periodic heartbeats to maintain authority, with the heartbeat mechanism checking if the last heartbeat time exceeds the interval before resending. The article includes detailed code showing how heartbeats are sent asynchronously to all peer nodes, with responses handled through CompletableFuture callbacks.

The Follower role is simpler, primarily monitoring heartbeat intervals and transitioning to Candidate if the leader becomes unresponsive for too long. The Candidate role involves the most complex logic, including term management, vote request handling, and consensus determination based on received responses.

The article discusses DLedger mode's limitations, including the requirement for at least three broker replicas, the need for majority-based acknowledgment protocols, and the existence of two separate HA replication flows that cannot leverage RocketMQ's native storage capabilities.

RocketMQ 5.0's Controller mode is presented as an improvement, introducing a standalone Raft consistency module (DLedger Controller) that can be deployed independently or embedded in Nameserver. The article compares the two deployment options: embedded in Nameserver versus independent deployment, discussing their respective advantages and disadvantages.

The embedded approach offers simplified deployment and reduced network communication but introduces resource competition and potential stability issues. Independent deployment provides better isolation and scalability but increases complexity and operational overhead.

The article concludes by emphasizing that while DLedger provides a solid foundation for leader election, practical implementation requires careful consideration of trade-offs in distributed consistency algorithms. It references key literature including Apache RocketMQ documentation and the Raft consensus algorithm paper.