RocketMQ Production Operations: Cluster Setup, Retry Mechanisms & Dead Letter Queue Solutions

Background and Applicable Scenarios

RocketMQ is Alibaba's open-source distributed messaging middleware, originally built to support Alibaba's Double 11 massive message throughput, now hosted by Apache Foundation. Compared to Kafka, RocketMQ excels in delayed messages, transaction messages, and flexible consumption patterns; compared to RabbitMQ, its master-slave architecture suits high-throughput horizontal scaling for medium-to-large distributed systems.

Typical use cases include async decoupling (order system offloads inventory deduction, points changes, logistics notifications), peak shaving (flash sales write requests to queue first), delayed tasks (order timeout cancellation, appointment reminders), distributed transactions (replacing two-phase commit), log collection/analysis (as log bus for ELK), and big data pipelines (real-time source for Flink/Spark Streaming).

Cluster Architecture and Core Concepts

Core Roles

NameServer : Service discovery & routing metadata management, stateless. Process: mqnamesrv Broker : Message storage & forwarding, persistence and replication. Process: mqbroker Producer : Message producer, sends to Broker. In-app process.

Consumer : Message consumer, pulls from Broker. In-app process.

NameServers don't communicate with each other; each maintains full routing info. Brokers register to all NameServers on startup; Producers/Consumers fetch Broker addresses via NameServer.

NameServer Design Trade-offs

Stateless enables horizontal scaling but only eventual consistency (Broker registration expires after 30s heartbeat)

Memory limited; each Broker carries massive Topic/Queue info — consider cluster split beyond 100k Topics

NameServer restart loses routing info; relies on Broker re-registration (~30s recovery)

Broker Key Responsibilities

Receive Producer messages, write to CommitLog

Maintain ConsumeQueue (consumption progress) and IndexFile (key/time query index)

Handle Consumer pull requests

Master-slave mode: Slave replication, Leader election (Dledger mode)

Periodic cleanup of expired files (default 4 AM, by fileReservedTime)

Deployment Mode Comparison

Single Master : Low cost, medium performance, low reliability. Local development.

Multi Master : Medium cost, high performance, medium reliability. Internal business, tolerates minor loss.

Multi Master + Multi Slave (Async) : High cost, high performance, medium reliability. High-throughput internal business.

Multi Master + Multi Slave (Sync) : High cost, medium performance, high reliability. Finance, orders — strong reliability.

Dledger (Recommended for Production) : Medium cost, medium performance, high reliability. Recommended for production.

Dledger + Cross-DC : Very high cost, medium performance, very high reliability. Geo-disaster recovery.

Article demonstrates Multi Master mode (2 NameServer + 2 Broker Master) covering most internal systems; production HA should switch to Dledger.

Message Storage Model

Core file structure under ${storePathRootDir}:

commitlog/          # Physical message storage, 1GB per file, filename = start offset
consumequeue/       # Consumption queue per Topic-Queue, 20-byte entries (8B offset + 4B size + 8B tag hash)
index/              # Index files for key/time query (not enabled by default)
abort/              # Abnormal termination marker (startup check)
checkpoint/         # Flush checkpoint
config/             # Broker runtime config

CommitLog : Sequential append of all Topics' messages; mmap for performance; file naming = 20-digit zero-padded start offset.

ConsumeQueue : Logical index per Topic+QueueId; 300k entries per file (~5.72MB).

IndexFile : 40B header + 5M IndexUnits (28B each) for key/timestamp queries.

Flush Disk Mechanism

ASYNC_FLUSH

: Write to PageCache, return SUCCESS immediately; background thread flushes every 500ms. High throughput but messages in PageCache lost on crash. SYNC_FLUSH: Write to PageCache then fsync before returning SUCCESS. ~30% throughput loss but no message loss on single-node crash.

Important: ASYNC_FLUSH is single-node reliability; replica sync is separate ( brokerRole ). Production recommends SYNC_MASTER + SYNC_FLUSH or Dledger + ASYNC_FLUSH (Dledger's Raft compensates async flush window).

Consumption Modes

Clustering (default) : Consumer Group instances share load; each message consumed once per group; progress stored on Broker. Suits long-processing, horizontally scalable scenarios.

Broadcasting : Every instance in group consumes full messages; progress stored locally. Suits cache updates, event distribution, config push.

Ordered Messages : Global order (single Queue) or Partition order (via MessageQueueSelector routing by business key). Critical: Ordered consumption failures retry infinitely — never enter DLQ (common misconception).

Message Filtering : Tag filtering (Broker-side hash) > SQL92 filtering ( MessageSelector.bySql()); complex filters better handled on Consumer side.

Core Concept Relationships

Topic: Logical message classification
Message Queue: Physical partition of Topic; multiple Queues increase parallelism
Consumer Group: Consumers share load; one message consumed once per group
Tag: Second-level classification under Topic for fine-grained filtering
Offset: Consumer's consumption position record
CommitLog: Broker's actual message log files

Queue count design: Write parallelism = writeQueueNums; Read parallelism = min( readQueueNums, Consumer instances). Typically read/write queues equal; Queue count = 2 × Broker nodes (1x headroom); 8-16 queues per Topic common.

Cluster Setup Practice (2 NameServer + 2 Broker Master)

Environment Preparation

Hosts: rocketmq-ns1 (192.168.1.101, NameServer-1), rocketmq-ns2 (192.168.1.102, NameServer-2), rocketmq-broker1 (192.168.1.103, Broker-Master-1), rocketmq-broker2 (192.168.1.104, Broker-Master-2)

Requirements: CentOS 7/Rocky 8/Ubuntu 20.04; JDK 8 or 11 (4.9.x); dedicated data disk (SSD recommended); NameServer 2GB+ RAM, Broker 8GB+; 1Gbps+ internal network, cross-AZ RTT ≤5ms.

Disk planning: Mount separate data disk to /data (XFS), create /data/rocketmq/{store,logs,config,backup} with subdirs for commitlog, consumequeue, index, abort, checkpoint. Symlink /usr/local/rocketmq/store and /logs to data disk. Create rocketmq user, chown directories.

Capacity estimation (reference): 100M msgs/day × 1KB = 100GB/day; 3-day retention = 300GB; + indexes/queues ≈ 330GB; reserve 50% → 500GB+ per Broker; SSD with IOPS ≥5000.

Installation & Directory Layout

cd /opt
wget https://archive.apache.org/dist/rocketmq/4.9.4/rocketmq-all-4.9.4-bin-release.zip
unzip ...
mv rocketmq-all-4.9.4-bin-release /usr/local/rocketmq
mkdir -p /data/rocketmq/store/{commitlog,consumequeue,index,abort,checkpoint}
mkdir -p /data/rocketmq/{config,logs}
ln -s /data/rocketmq/store /usr/local/rocketmq/store
ln -s /data/rocketmq/logs /usr/local/rocketmq/logs
useradd -r -s /sbin/nologin rocketmq
chown -R rocketmq:rocketmq /usr/local/rocketmq /data/rocketmq

Risk: Never run Broker/NameServer as root. 4.x warns but runs; 5.x enforces non-root.

NameServer Configuration & Startup

Config conf/nameserver.conf minimal; default listenPort=9876 usually fine.

systemd unit ( /etc/systemd/system/rocketmq-nameserver.service):

[Unit]
Description=RocketMQ NameServer
After=network.target

[Service]
Type=simple
User=rocketmq
Environment="JAVA_OPT_EXT=-server -Xms2g -Xmx2g -Xmn1g"
ExecStart=/usr/local/rocketmq/bin/mqnamesrv
ExecStop=/usr/local/rocketmq/bin/mqshutdown namesrv
Restart=always
RestartSec=10
StandardOutput=null
StandardError=/data/rocketmq/logs/nameserver.log

[Install]
WantedBy=multi-user.target

Enable/start on both nodes; verify with ss -tlnp | grep 9876. Common failures: port conflict, JDK mismatch (4.x needs 8/11), permission issues, insufficient heap memory.

Broker Configuration & Startup

Base config conf/broker.conf; create per-node configs.

Broker Master 1 (192.168.1.103):

brokerClusterName=rocketmq-cluster
brokerName=broker-a
brokerId=0
listenPort=10911
haListenPort=10912
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
storePathRootDir=/usr/local/rocketmq/store
storePathCommitLog=/usr/local/rocketmq/store/commitlog
storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue
storePathIndex=/usr/local/rocketmq/store/index
storeCheckpoint=/usr/local/rocketmq/store/checkpoint
abortFile=/usr/local/rocketmq/store/abort
fileReservedTime=72
deleteWhen=04
maxMessageSize=524288
autoCreateTopicEnable=true
brokerIP1=192.168.1.103

Broker Master 2 (192.168.1.104): Same but brokerName=broker-b, brokerIP1=192.168.1.104.

Key brokerIP1 : Must be reachable IP for Producers/Consumers; if multi-NIC, set to accessible address.

JVM tuning: Default runbroker.sh uses 8GB heap; adjust to ≤50% physical RAM (demo uses 2GB).

systemd unit ( /etc/systemd/system/rocketmq-broker.service):

[Unit]
Description=RocketMQ Broker
After=network.target

[Service]
Type=simple
User=rocketmq
Environment="JAVA_OPT_EXT=-server -Xms2g -Xmx2g -Xmn1g"
ExecStart=/usr/local/rocketmq/bin/mqbroker \
  -c /usr/local/rocketmq/conf/broker-master1.conf \
  -n "192.168.1.101:9876;192.168.1.102:9876"
ExecStop=/usr/local/rocketmq/bin/mqshutdown broker
Restart=always
RestartSec=10
StandardOutput=null
StandardError=/data/rocketmq/logs/broker.log

[Install]
WantedBy=multi-user.target

Risk: -n must list ALL NameServer addresses (semicolon-separated). Single NameServer registration breaks HA.

Multi Master + Multi Slave Extension

Add Slave nodes with brokerId=1 (non-zero), brokerRole=SLAVE, same brokerName as Master.

Dledger Mode Quick Switch

Dledger (4.5.0+) uses Raft for auto leader election. Config example for 3-node broker-a group:

enableDLegerCommitLog=true
dLegerGroup=broker-a
dLegerPeers=n0-192.168.1.103:40911;n1-192.168.1.104:40911;n2-192.168.1.105:40911
dLegerSelfId=n0
flushDiskType=ASYNC_FLUSH  # Raft ensures replica consistency

All nodes brokerId=0; start simultaneously to avoid prolonged election.

Dledger vs Multi Master+Slave: Dledger auto-failover (Raft), manual for traditional; Dledger ~20% lower write throughput (majority ACK); sync dual-write outperforms Dledger by ~20%.

Dashboard Deployment (Optional but Recommended)

Download rocketmq-dashboard-1.0.0.jar, run on port 8080 with -Drocketmq.config.namesrvAddr=.... Access via http://192.168.1.101:8080. Secure with Nginx auth or internal-only access (no built-in auth).

Cluster Verification

Create test Topic:

mqadmin updateTopic -n $NAMESRV_ADDR -t TestTopic -c rocketmq-cluster -r 8 -w 8

Send test messages: tools.sh org.apache.rocketmq.example.quickstart.Producer (sends 1000 msgs)

Consume test: tools.sh org.apache.rocketmq.example.quickstart.Consumer (expect CONSUME OK)

Dashboard check: Cluster page shows both Brokers UP; Topic queues evenly distributed; Consumer group online.

Message Send & Consume Mechanisms

Send Modes

Sync : Wait for Broker ACK; important notifications.

Async : Non-blocking with SendCallback; high concurrency.

Oneway : Fire-and-forget; max throughput but possible loss; logging scenarios.

Ordered : Use MessageQueueSelector with business key (e.g., orderId) to fix Queue.

Transaction : Half-message → local transaction → Commit/Rollback → Broker back-check if timeout.

Consume Modes

Clustering : MessageListenerConcurrently; return SUCCESS or RECONSUME_LATER.

Broadcasting : MessageListenerConcurrently with MessageModel.BROADCASTING.

Ordered : MessageListenerOrderly; auto-locks per Queue; failure returns SUSPEND_CURRENT_QUEUE_A_MOMENT.

Message Retry Mechanism Deep Dive

Retry Trigger Conditions

Producer-side (sync) : Default 2 retries ( retryTimesWhenSendFailed); triggers on timeout/ RemotingException; NOT on business exceptions or MQClientException.

Consumer-side : Uncaught exception; return RECONSUME_LATER; consumption timeout (default 15min, configurable consumeTimeout).

Retry Count & Interval

Default 16 retries for non-ordered messages; then DLQ. Backoff schedule (default):

Retry # | Interval | Cumulative
1       | 10s      | 10s
2       | 30s      | 40s
3       | 1m       | 1m40s
4       | 2m       | 3m40s
5       | 3m       | 6m40s
6       | 4m       | 10m40s
7       | 5m       | 15m40s
8       | 6m       | 21m40s
9       | 7m       | 28m40s
10      | 8m       | 36m40s
11      | 9m       | 45m40s
12      | 10m      | 55m40s
13      | 20m      | 1h15m40s
14      | 30m      | 1h45m40s
15      | 1h       | 2h45m40s
16      | 2h       | 4h45m40s

Intervals derived from messageDelayLevel (1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h). Retry N uses delayLevel N+2. Verify via mqbroker -p or source MessageConst .

Custom max retries: Consumer setMaxReconsumeTimes(3) or Broker maxReconsumeTimes=16. Risk: Too small → normal fluctuations go DLQ; too large (default 16) → downstream outage piles DLQ. Set per downstream SLA (3-5 typical).

Retry Topic Transformation

Retried messages move to %RETRY%{consumerGroup} (not original Topic). Consumer transparently pulls from there. Special properties: RETRY_TOPIC (original Topic), ORIGIN_MESSAGE_ID, RECONSUME_TIME (retry count), DLQ_NEXT_CONSUME_TIME (next retry timestamp).

Handling advice: Differentiate by retry count (alert after 5+); don't blanket-retry all exceptions — distinguish transient (DB connection) vs permanent (data format); permanent should go DLQ directly.

Ordered Message Retry Special Handling

Ordered consumption failures never enter DLQ — infinite retry. Risk: single poison message blocks entire Queue. Mitigation: try-catch in business logic; for permanent errors, log and force SUCCESS (if loss acceptable) or forward to compensation Topic.

Practical: Controlling Retry Rhythm

Scenario: DB write failure → catch SQLException retry, DataFormatException → log + SUCCESS (or custom DLQ Topic).

Scenario: Third-party rate limit → set msg.setDelayTimeLevel(5) (1min delay) on re-send.

Scenario: Retried 1 hour, don't want more but can't lose → check msg.getReconsumeTimes() >= 5, actively write to custom DLQ Topic, then continue.

Dead Letter Queue (DLQ) Practice

DLQ Generation Rules

Non-ordered messages exceeding max retries (default 16) enter %DLQ%{consumerGroup} (not original Topic). Key properties: RETRY_TOPIC, ORIGIN_MESSAGE_ID, ORIGIN_QUEUE_ID, RECONSUME_TIME (16), DLQ_NEXT_CONSUME_TIME (timestamp), SHARDING_KEY (for ordered).

Query via mqadmin queryMsgById -n $NAMESRV_ADDR -msgId ....

DLQ Processing Flow

Consume fail → Retry (N times) → Exceed max retries → Enter %DLQ%{group}
  → Ops detects via monitoring/alert → Manual analysis of root cause
  → Decision: Re-send / Manual compensation / Record & discard
  → Fix downstream → Clean DLQ

DLQ Inspection Commands

# List DLQ topics
mqadmin topicList -n $NAMESRV_ADDR | grep %DLQ%

# Query by key
mqadmin queryMsgByKey -n $NAMESRV_ADDR -t %DLQ%my-group -k msgId

# Query by offset
mqadmin queryMsgByOffset -n $NAMESRV_ADDR -t %DLQ%my-group -b broker-a -i 0 -o 0

# Time-range query (ms timestamps)
START_TS=$(date -d "2024-01-15 00:00:00" +%s)000
END_TS=$(date -d "2024-01-16 00:00:00" +%s)000
mqadmin printMsg -n $NAMESRV_ADDR -t %DLQ%my-group -s "$START_TS" -e "$END_TS" --printBody true

DLQ Reprocessing Practice

Solution 1 (Recommended): One-off Consumer reads %DLQ%{group}, extracts RETRY_TOPIC property, re-sends body to original Topic.

// Pseudocode: consume from DLQ, get originalTopic from RETRY_TOPIC property,
// create new Message(originalTopic, tags, body), send via Producer.

Solution 2: Dashboard UI (test env only). Risk: DLQ messages never auto-expire — monitor/alert and clean regularly. Test reprocessing scripts in staging first.

DLQ Alerting Script

#!/bin/bash
NAMESRV_ADDR="192.168.1.101:9876;192.168.1.102:9876"
CONSUMER_GROUP="my-consumer-group"
DLQ_TOPIC="%DLQ%${CONSUMER_GROUP}"
THRESHOLD=10
dlq_count=$(mqadmin topicStatus -n "$NAMESRV_ADDR" -t "$DLQ_TOPIC" 2>/dev/null | awk 'NR>1 {sum+=$2} END {print sum+0}')
if [ "$dlq_count" -gt "$THRESHOLD" ]; then
  echo "WARNING: DLQ has $dlq_count messages (threshold=$THRESHOLD)"
  curl -X POST "https://alert.example.com/hook" -H "Content-Type: application/json" -d "{\"msg\": \"RocketMQ DLQ alert: $CONSUMER_GROUP has $dlq_count dead letters\"}"
  exit 1
else
  echo "OK: $dlq_count dead letters in $CONSUMER_GROUP"
  exit 0
fi

Cron:

*/5 * * * * /usr/local/scripts/check_dlq.sh >> /var/log/rocketmq_dlq_check.log 2>&1

Operations & Troubleshooting

Message Accumulation Investigation

Symptom: Consumer group lag ( diffTotal) growing.

Path: 1) Confirm accumulation via Dashboard; 2) Check Consumer alive ( ps aux | grep consumer, logs for RECONSUME_LATER, OOM, GC pauses); 3) Capacity match: Consumer count < Queue count? Duplicate registrations? Single-message processing latency?; 4) Broker bottlenecks: CPU/memory/disk IO, PageCache hit rate, network packet loss.

Key commands: mqadmin consumerStatus -g group (watch diffTotal, JVM memory, threads); mqadmin consumerProgress -g group (per-queue offsets); df -h /store, du -sh commitlog; ss -s, netstat -anp | grep 10911 | wc -l.

SOP: Temporarily scale Consumers to Queue count; identify slow messages via Dashboard/tracing; optimize business logic (async, batch, RPC timeouts); reset offset via resetOffsetByTime if messages droppable; consider Topic split if single Topic too large.

Consume Latency Diagnosis

Latency = consume rate < produce rate; accumulation = backlog. Additional causes:

Consumer GC pauses : Frequent Young/Full GC → increase heap, reduce object allocation, upgrade to G1/ZGC.

Thread pool misconfig : Slow logic (RPC/DB) needs more threads: setConsumeThreadMin(20), setConsumeThreadMax(40).

Retry storms : One message repeatedly retries blocking Consumer instance; locate via queryMsgById or consumerStatus / consumerProgress.

Pull interval : Too short → Broker hammered; too long → idle. Tune setPullInterval(1000) (ms), setPullBatchSize(32).

Broker Node Failure Emergency

systemctl restart rocketmq-broker

. If disk full → clean then restart. Emergency cleanup: snapshot cp -r store /backup/store_$(date +%Y%m%d%H%M%S), then find commitlog -name "*.log" -mtime +3 -delete.

Multi Master failure impact:

Broker Master-1 down: Its Queues unwritable; existing messages readable by Slave (if any)

Both Masters down: Cluster unwritable (even with Slaves); emergency restore or failover to backup cluster

One NameServer down: Cluster OK (Broker registers to other); Producers/Consumers need restart/retry

All NameServers down: Existing connections unaffected; new Producers/Consumers cannot connect

Risk: Multi Master mode loses in-flight messages on Master crash. Production must use Dledger for auto-failover.

Dledger leader check: mqadmin dledgerGetLeader -k broker-a; force switch (maintenance only): mqadmin dledgerElectLeader -k broker-a -id n1.

Disk Full Handling

Alert: DISK_USAGE_WARNING_RATIO=0.90, used=92.34%.

Steps: 1) df -h, du -sh store/*; 2) Clean logs ( find logs -name "*.log.*" -mtime +7 -delete, find logs -name "*.log" -size +1G -mtime +3 -delete); 3) Trigger commitlog cleanup (restart Broker or Dashboard); 4) Temporarily lower fileReservedTime from 72 to 24 (restart required); 5) Expand disk (ultimate fix). Never rm -rf commitlog on running Broker — causes message loss.

Broker OOM & GC Tuning

Symptoms: OOM Killer, Full GC >1/min, slow responses, send timeouts.

Diagnose: ps -p $(pgrep -f mqbroker) -o pid,rss,vsz,cmd (RSS = physical); tail -f gc-broker.log (Full GC frequency/duration).

GC config in runbroker.sh:

JAVA_OPT="${JAVA_OPT} -server \
  -Xms8g -Xmx8g -Xmn4g \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:InitiatingHeapOccupancyPercent=45 \
  -XX:+ParallelRefProcEnabled \
  -XX:+PrintGCDetails -XX:+PrintGCDateStamps \
  -XX:GCLogFileSize=100M -XX:NumberOfGCLogFiles=10 \
  -Xlog:gc*:file=/data/rocketmq/logs/gc-broker.log:time"

Principles: Xms=Xmx; NewRatio=2 (1/3 young, 2/3 old); G1 target pause 100-200ms; IHOP 45%; Monitor: Young GC <1/sec, Full GC <1/day.

Consume Progress Reset

Reset to timestamp:

mqadmin resetOffsetByTime -n $NAMESRV_ADDR -g group -t Topic -s "2024-01-01 10:00:00" -f true

.

Reset to latest (skip backlog):

mqadmin resetOffsetByTime -n $NAMESRV_ADDR -g group -t Topic -s "$(date +%Y-%m-%d\ %H:%M:%S)" -f true

.

Risk: Skips unprocessed messages → data loss/inconsistency. Business must confirm impact; operate off-peak; always backup offsets first : mqadmin consumerProgress -g group > /backup/consumerProgress_$(date +%Y%m%d%H%M%S).txt .

Monitoring & Alerting

Prometheus Integration

Deploy rocketmq-exporter (port 5557) with --rocketmq.config.namesrvAddr=.... Prometheus scrape config: job rocketmq, target exporter:5557, interval 15s.

Key Metrics

Broker: rocketmq_broker_tps (TPS, alert 2× baseline), rocketmq_queue_size (backlog >10k warn), rocketmq_disk_usage (>80% warn, >90% critical), rocketmq_commitlog_min/max_offset.

Consumer: rocketmq_consumer_lag (>10k warn), rocketmq_consumer_throughput (<0.5× baseline), rocketmq_consumer_reconsume_times (>3 warn).

Producer: rocketmq_producer_throughput (2× baseline), rocketmq_producer_latency (>100ms warn), rocketmq_producer_fail_count (>0 immediate alert).

Verify actual exporter metrics via curl http://exporter:5557/metrics . Thresholds must align with business baselines (e.g., Double 11 lag thresholds differ). Use time-based threshold groups.

Alert Rules Example (Prometheus)

groups:
- name: rocketmq
  rules:
  - alert: RocketMQBrokerDown
    expr: rocketmq_cluster_broker_up == 0
    for: 1m
    labels: {severity: critical}
    annotations: {summary: "RocketMQ Broker down", description: "Broker {{ $labels.broker }} down for 1m"}
  - alert: RocketMQConsumerLag
    expr: rocketmq_consumer_lag > 10000
    for: 5m
    labels: {severity: warning}
    annotations: {summary: "Consumer {{ $labels.group }} lag high", description: "Lag {{ $value }}, check consumer"}
  - alert: RocketMQDiskUsageHigh
    expr: (rocketmq_disk_usage / 100) > 0.85
    for: 5m
    labels: {severity: warning}
    annotations: {summary: "Broker disk usage high", description: "Usage {{ $value }}%, clean up"}
  - alert: RocketMQDLQHasMessage
    expr: rocketmq_dlq_message_count > 0
    for: 1m
    labels: {severity: warning}
    annotations: {summary: "DLQ has messages", description: "Consumer Group {{ $labels.group }} has dead letters"}
  - alert: RocketMQBrokerFullGC
    expr: rate(rocketmq_jvm_gc_count{type="full"}[5m]) > 0.01
    for: 5m
    labels: {severity: warning}
    annotations: {summary: "Broker Full GC frequent", description: "{{ $labels.broker }} Full GC {{ $value }}/sec over 5m"}

Risk Warnings & Best Practices

Production Considerations

HA Architecture: Minimum Multi Master + monitoring; Standard Dledger (auto-failover); High Dledger multi-replica + multi-DC DR.

Topic Planning: Domain-based separation; Queue count = 2 × Master nodes; Disable autoCreateTopicEnable; Naming convention {domain}_{purpose}_{env} (e.g., order_created_prod).

Consumer Design: Instances ≤ Queue count; Idempotent consumption (duplicates possible); Avoid long ops in callback (async); Try-catch in callback to prevent instance crash.

Producer Design: Sync send timeout 5-10s; Async must implement SendCallback; Important messages enable setRetryAnotherBrokerWhenNotStoreOK; Message keys well-distributed to avoid hot Queue.

Monitoring System: Broker: CPU, mem, disk, commitlog write latency, GC; Consumer: lag, TPS, retry count, DLQ count; Producer: success rate, latency, timeouts.

Log Collection: Centralize GC, Broker, consumer logs to ELK/Loki; Alert keywords: ERROR, Exception, RECONSUME_LATER, flush disk, broker busy; Retain 30 days.

Canary Release & Rollback

Topic canary: Create Topic_gray → route partial Producer traffic (user ID hash) → gray Consumer subscribes → observe 1-2 days → cut back to main Topic → retain gray Consumer 1 week.

Broker upgrade rollback SOP: Backup /usr/local/rocketmq and /store; Stop new Broker; Install old version over /usr/local/rocketmq; Start; Verify via mqadmin brokerStatus. Major version upgrades (4.x→5.x) not fully compatible; test thoroughly. Downgrade 5.x→4.x usually impossible due to commitlog format changes.

Backup Strategy

NameServer stateless — restart recovers. Broker data on local disk. Daily backup script (3 AM):

#!/bin/bash
DATE=$(date +%Y%m%d)
BACKUP_DIR=/data/backup/rocketmq
mkdir -p $BACKUP_DIR/$DATE
cp -r /usr/local/rocketmq/conf $BACKUP_DIR/$DATE/
# Optional store backup (exclude huge commitlog/consumequeue)
tar -czf $BACKUP_DIR/$DATE/store_$DATE.tar.gz -C /usr/local/rocketmq/store --exclude=commitlog --exclude=consumequeue config checkpoint abort
mqadmin consumerProgress -n $NAMESRV_ADDR -g my-group > $BACKUP_DIR/$DATE/consumerProgress.txt
find $BACKUP_DIR -maxdepth 1 -type d -mtime +30 -exec rm -rf {} \;

Restore: Unpack RocketMQ on new machine → restore conf → if store restored, start Broker (auto-loads commitlog) → verify brokerStatus → check consume progress, adjust with resetOffsetByTime if needed.

Security Hardening

Network: 9876/10911 internal only; firewall/security-group restrict source IPs; Dashboard 8080 behind Nginx auth or internal-only.

Auth (4.4.0+): aclEnable=true in broker.conf; Clients use AK/SK.

TLS: 4.4.0+ supports SSL/TLS (cert config required).

Audit: Enable access logs; Audit mqadmin calls (updateTopic, deleteTopic, resetOffsetByTime, updateBrokerConfig).

Capacity Planning Checklist

☐ Topic count forecast (1-3 months)

☐ Per-Topic queue count design

☐ Max message size

☐ Daily peak message volume

☐ Disk capacity (fileReservedTime × peak traffic)

☐ Broker nodes (≥2 Masters + full replicas)

☐ Network bandwidth (≥1Gbps internal, cross-DC dedicated line)

☐ Monitoring alerts (Broker, Consumer, Producer metrics)

☐ Backup strategy (config, offsets, commitlog incremental)

☐ Canary process (Topic gray, Broker gray)

☐ Emergency plans (disk full, Broker crash, accumulation, DLQ)

Summary

RocketMQ operations core: HA architecture (no single point in NameServer/Broker), Message reliability (flush policy, retry, DLQ), Observability (monitoring, alerting, logging).

Retry & DLQ are last line of defense: Retry per scenario (transient retry, permanent → DLQ); DLQ not a trash bin — establish regular reprocessing/compensation flow; Limit retries (3-5) to avoid DLQ pile-up during outages; Ordered messages lack DLQ exit — business must handle.

Daily Ops SOP: 1. Accumulation → Consumer alive? → Capacity match? → Broker bottleneck? 2. Broker crash → Restart → Check replica → Failover to backup cluster if needed 3. Disk full → Clean logs → Lower retention → Expand disk 4. DLQ pile → Extract & analyze root cause → Fix → Reprocess → Clean

One-liner: Master mqadmin CLI — it's the fundamental ops tool; Dashboard aids daily efficiency but CLI is first-responder at incident scenes. Build habit of checking logs and command output first, don't over-rely on UI.

Appendix A: Common mqadmin Commands Cheatsheet

Cluster status: mqadmin clusterList -n $NAMESRV_ADDR Topic list: mqadmin topicList -n $NAMESRV_ADDR Topic route: mqadmin topicRouteList -n $NAMESRV_ADDR -t $TOPIC Create Topic:

mqadmin updateTopic -n $NAMESRV_ADDR -t $TOPIC -c $CLUSTER -r 8 -w 8

Delete Topic: mqadmin deleteTopic -n $NAMESRV_ADDR -t $TOPIC -c $CLUSTER Consume progress: mqadmin consumerProgress -n $NAMESRV_ADDR -g $GROUP Consumer group list: mqadmin consumerList -n $NAMESRV_ADDR Consumer connections: mqadmin consumerConnectionSubList -n $NAMESRV_ADDR -g $GROUP Consumer status: mqadmin consumerStatus -n $NAMESRV_ADDR -g $GROUP Reset offset:

mqadmin resetOffsetByTime -n $NAMESRV_ADDR -g $GROUP -t $TOPIC -s $TIME -f true

View DLQ: mqadmin topicStatus -n $NAMESRV_ADDR -t %DLQ%$GROUP Query by msgId: mqadmin queryMsgById -n $NAMESRV_ADDR -msgId $MSGID Query by key: mqadmin queryMsgByKey -n $NAMESRV_ADDR -t $TOPIC -k $KEY Query by offset:

mqadmin queryMsgByOffset -n $NAMESRV_ADDR -t $TOPIC -b $BROKER -i $QUEUE_ID -o $OFFSET

Broker status: mqadmin brokerStatus -n $NAMESRV_ADDR -b $BROKER_NAME View retry queue: mqadmin topicStatus -n $NAMESRV_ADDR -t %RETRY%$GROUP Dledger status:

mqadmin dledgerGetLeader -n $NAMESRV_ADDR -k $DLER_GROUP

Appendix B: Common Fault Quick Reference

Send timeout → NS/Broker port connectivity → Restart Broker

Lag growing → Consumer process / GC / slow logic → Scale Consumers temporarily

Disk >90% → CommitLog growth rate → Clean old logs, lower fileReservedTime

DLQ pileup → Business consume logic → Fix root cause, then reprocess

Frequent Full GC → Heap config → Adjust Xmx, tune G1 params

NS restart → stale routes → Broker registration → Wait 30s for auto re-register

Dledger election fail → Majority nodes unreachable → Check network, restore ≥ half nodes

Ordered message stuck → Single message processing fail → Business code fallback skip

Dashboard inaccessible → Dashboard process/port → Restart Dashboard

Auto-create Topic fail → autoCreateTopicEnable setting → Manually create Topic

Appendix C: Version Notes

RocketMQ 4.9.x (4.9.4 used here): Stable production; JDK 8/11; Mature Dledger; Complete ACL; Widely used in enterprise.

RocketMQ 5.x: New architecture (Controller + Raft); Pop consumption model; Better elastic scaling; Partial config incompatibility with 4.x; Upgrade requires full test validation.

Client Version: Match Broker major version; Spring Boot use rocketmq-spring-boot-starter; Multi-language clients (Go, Python, C++, Node.js) prefer official maintained versions. Fields may vary by version — always verify against deployed version docs. Test full chain in staging before any upgrade.