Building a High‑Availability RocketMQ Cluster (Version 4.9.4)

Hello everyone, I am Chen.

RocketMQ, an open‑source MQ middleware from the Alibaba ecosystem, has proven its ability to handle trillion‑level messages under high‑concurrency scenarios such as Double‑Eleven.

This article is the first in the "RocketMQ Advanced" series and demonstrates how to build a highly available RocketMQ cluster in production.

Why Use a Message Queue?

Message queues provide a FIFO data structure and address three main scenarios:

1. Application Decoupling

High coupling reduces fault tolerance. In e‑commerce, if order creation directly calls inventory, logistics, and payment services, any failure blocks the whole order flow. Using a queue buffers requests, allowing the front‑end to succeed while downstream services recover.

2. Traffic Shaping (Peak‑Smoothing)

During traffic spikes, a queue can buffer massive requests and release them gradually, improving system stability and user experience.

From an economic perspective, provisioning enough servers for peak QPS is often uneconomical; a queue helps smooth peaks without over‑provisioning.

3. Data Distribution

Producers publish messages without needing to know consumers; consumers pull data directly from the queue, enabling flexible data flow between systems.

Comparison of Common MQ Products

Popular options include Kafka, ActiveMQ, RabbitMQ, and RocketMQ. (See the linked article for detailed selection criteria.)

Key Roles in RocketMQ

NameServer : Service discovery for brokers; nodes do not communicate with each other.

Producer : Sends messages to brokers; producers sharing a groupId form a cluster.

Broker : Stores and forwards messages using push/pull mechanisms.

Consumer : Retrieves messages from brokers; consumers sharing a groupId form a cluster.

Internal concepts:

Topic : Logical grouping of messages.

Tag : Fine‑grained label within a Topic.

Message Queue : Partition of a Topic for parallel processing.

Downloading RocketMQ

RocketMQ is contributed to Apache; download the desired version from the Apache site. This tutorial uses version 4.9.4 (download URL: https://rocketmq.apache.org/download).

Cluster Characteristics

NameServer : No inter‑node communication; simply start multiple instances.

Broker : Deployed as master‑slave pairs; masterId=0, slaveId=1; each broker registers topics with NameServer.

Producer : Connects to any NameServer, fetches routing info, and maintains a long‑lived connection to the master serving the target Topic; stateless and cluster‑ready.

Consumer : Randomly selects a NameServer, obtains routing, and connects to master or slave based on broker configuration.

Cluster Modes

Four broker cluster modes are supported:

1. Single‑Master

Not recommended; a single point of failure.

2. Multi‑Master (no slaves)

All nodes are masters; highest performance but any master failure makes its messages unavailable.

3. Multi‑Master + Multi‑Slave (Synchronous)

Each master has a synchronous slave; a message is acknowledged only after the slave replicates the data, ensuring no data loss at the cost of lower throughput.

4. Multi‑Master + Multi‑Slave (Asynchronous)

Replication is asynchronous; higher throughput with minimal latency, still providing fault tolerance.

Building a Synchronous Master‑Slave Cluster

The setup uses four nodes: two masters and two slaves, distributed across two machines.

1. Prepare Environment

Two servers/VMs.

JDK 1.8 installed.

Download rocketmq-all-4.9.4-bin-release.

2. Add Environment Variables

export PATH=$JAVA_HOME/bin:$PATH
export ROCKETMQ_HOME=/usr/local/rocketmq-all-4.9.4-bin-release
export PATH=$PATH:$ROCKETMQ_HOME/bin

3. Create Storage Directories

mkdir -p /usr/local/rocketmq/store/master/commitlog
mkdir -p /usr/local/rocketmq/store/master/consumequeue
mkdir -p /usr/local/rocketmq/store/master/index
mkdir -p /usr/local/rocketmq/store/slave/commitlog
mkdir -p /usr/local/rocketmq/store/slave/consumequeue
mkdir -p /usr/local/rocketmq/store/slave/index

4. Broker Configuration Files

Four property files are required:

master1 (broker‑a.properties)

# Cluster name
brokerClusterName=rocketmq-cluster
# Broker name
brokerName=broker-a
# 0 = master
brokerId=0
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=300000
storePathRootDir=/usr/local/rocketmq/store
storePathCommitLog=/usr/local/rocketmq/store/commitlog
storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue
storePathIndex=/usr/local/rocketmq/store/index
maxMessageSize=65536
brokerRole=SYNC_MASTER
flushDiskType=SYNC_FLUSH

slave2 (broker‑b-s.properties)

# Cluster name
brokerClusterName=rocketmq-cluster
brokerName=broker-b
brokerId=1
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=11011
storePathRootDir=/usr/local/rocketmq/store/slave
storePathCommitLog=/usr/local/rocketmq/store/slave/commitlog
storePathConsumeQueue=/usr/local/rocketmq/store/slave/consumequeue
storePathIndex=/usr/local/rocketmq/store/slave/index
maxMessageSize=65536
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH

master2 (broker‑b.properties)

# Same as master1 but on the second machine
brokerClusterName=rocketmq-cluster
brokerName=broker-b
brokerId=0
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
storePathRootDir=/usr/local/rocketmq/store
storePathCommitLog=/usr/local/rocketmq/store/commitlog
storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue
storePathIndex=/usr/local/rocketmq/store/index
maxMessageSize=65536
brokerRole=SYNC_MASTER
flushDiskType=SYNC_FLUSH

slave1 (broker‑a-s.properties)

# Same as slave2 but for broker‑a
brokerClusterName=rocketmq-cluster
brokerName=broker-a
brokerId=1
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=11011
storePathRootDir=/usr/local/rocketmq/store/slave
storePathCommitLog=/usr/local/rocketmq/store/slave/commitlog
storePathConsumeQueue=/usr/local/rocketmq/store/slave/consumequeue
storePathIndex=/usr/local/rocketmq/store/slave/index
maxMessageSize=65536
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH

5. Open Required Ports

# Stop firewall (quick method)
systemctl stop firewalld.service
firewall-cmd --state
systemctl disable firewalld.service

Or open specific ports (9876 for NameServer, 10911 for master, 11011 for slave):

firewall-cmd --remove-port=9876/tcp --permanent
firewall-cmd --remove-port=10911/tcp --permanent
firewall-cmd --remove-port=11011/tcp --permanent
firewall-cmd --reload

6. Update /etc/hosts

# NameServer
192.168.47.146 rocketmq-nameserver1
192.168.47.145 rocketmq-nameserver2
# Broker
192.168.47.146 rocketmq-master1
192.168.47.146 rocketmq-slave2
192.168.47.145 rocketmq-master2
192.168.47.145 rocketmq-slave1

systemctl restart network

7. Adjust Startup Scripts

Modify runbroker.sh and runserver.sh in the bin directory to suit your server's memory.

8. Start Services

Start NameServer on both machines:

cd /usr/local/rocketmq-all-4.9.4-bin-release/bin
nohup sh mqnamesrv &

Start the four brokers:

# master1
nohup sh mqbroker -c ../conf/2m-2s-sync/broker-a.properties &
# slave2
nohup sh mqbroker -c ../conf/2m-2s-sync/broker-b-s.properties &
# master2
nohup sh mqbroker -c ../conf/2m-2s-sync/broker-b.properties &
# slave1
nohup sh mqbroker -c ../conf/2m-2s-sync/broker-a-s.properties &

9. Verify Processes and Logs

Check running processes and view logs:

# NameServer log
tail -500f ~/logs/rocketmqlogs/namesrv.log
# Broker log
tail -500f ~/logs/rocketmqlogs/broker.log

Cluster Monitoring Dashboard

Use the open‑source rocketmq-dashboard project. Update application.yml with your NameServer addresses, then build and run:

mvn clean package -Dmaven.test.skip=true
java -jar target/rocketmq-dashboard-1.0.1-SNAPSHOT.jar

Access the UI at http://<i>your‑ip</i>:8080 and view cluster status.

Conclusion

This article covered the fundamentals of MQ, introduced RocketMQ's architecture, and provided a complete step‑by‑step guide to deploying a synchronous master‑slave RocketMQ cluster; subsequent sections will explore message production and consumption configurations.