Understanding RocketMQ Message Tracing: Concepts, Configuration, and Custom TraceTopic Implementation

Introduction

Hello, I am the author. A friend was interviewed at JD and was asked about RocketMQ message tracing. The interview question was "What is message tracing?" and the interviewer added that it is configured on the broker node.

1. What is Message Tracing

RocketMQ's message tracing is a feature that tracks the full‑link status of a message from production, storage, to consumption, recording the complete lifecycle to help quickly locate issues such as loss, duplicate consumption, or delay.

Message tracing data key attributes

Producer side

Consumer side

Broker side

Producer instance info

Consumer instance info

Message Topic

Send time

Delivery time, delivery attempts

Message storage location

Send success flag

Consume success flag

Message Key

Send latency

Consume latency

Message Tag

Core purposes of message tracing

Problem diagnosis – quickly determine whether a message was sent, stored, consumed, and why it failed.

Link tracing – view the path and timing of a message across producer, broker, and consumer.

Status monitoring – aggregate metrics such as send/consume latency and success rates.

Data audit – retain the full lifecycle for compliance.

2. Message Tracing Configuration

To enable tracing, add the following property:

traceTopicEnable=true

Broker‑side properties (excerpt from the official documentation) include:

brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=0
deleteWhen=04
fileReservedTime=48
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
storePathRootDir=/data/rocketmq/rootdir-a-m
storePathCommitLog=/data/rocketmq/commitlog-a-m
autoCreateSubscriptionGroup=true
## if msg tracing is open, the flag will be true
traceTopicEnable=true
listenPort=10911
brokerIP1=XX.XX.XX.XX1
namesrvAddr=XX.XX.XX.XX:9876

RocketMQ supports two deployment modes for tracing:

Normal mode – tracing data shares the same broker nodes as regular business messages; simple to deploy but may cause IO contention under high throughput.

Physical IO isolation mode – tracing data is stored on dedicated broker nodes, avoiding resource competition and improving stability for large‑scale tracing.

3. Implementation Principle

Tracing works via:

Built‑in trace Topic (default RMQ_SYS_TRACE_TOPIC ) for storing trace records.

Hook mechanism – producers and consumers intercept send/consume events, generate trace data, and forward it to the trace Topic.

Broker stores the trace Topic; users can query via the RocketMQ console or API.

Workflow summary:

Producer: send business message → hook records trace → async report → store in trace Topic.

Consumer: consume message → hook records trace → async report → store in trace Topic.

All trace records are consolidated in RMQ_SYS_TRACE_TOPIC for later inspection.

4. Trace Topic Support Modes

Two ways to store trace data:

Default built‑in trace Topic ( RMQ_SYS_TRACE_TOPIC ), automatically created by RocketMQ.

Custom trace Topic defined by the user (e.g., MyApp_TraceTopic ) for special storage or security requirements.

4.1 Default Built‑in Trace Topic

Name is RMQ_SYS_TRACE_TOPIC , created automatically.

Enabled by default when producers/consumers set setEnableMsgTrace(true) .

Broker manages queues and storage automatically.

Suitable for quick validation or test environments.

4.2 Custom Trace Topic

Users can specify a custom Topic name such as MyApp_TraceTopic to store trace data.

Useful when trace data needs isolated clusters, sharding, or stricter security.

5. Custom TraceTopic Code Practice

5.1 Producer with Custom TraceTopic

import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.remoting.common.RemotingHelper;
public class CustomTraceProducer {
public static void main(String[] args) throws Exception {
// 1. Create producer, enable tracing, specify custom trace topic
DefaultMQProducer producer = new DefaultMQProducer("OrderProducerGroup", true, "MyApp_TraceTopic");
producer.setNamesrvAddr("localhost:9876");
producer.start();
try {
// 2. Build business message
Message msg = new Message("OrderTopic", "PayOrder", "Order_202307280001", "订单支付成功".getBytes(RemotingHelper.DEFAULT_CHARSET));
// 3. Send message
SendResult sendResult = producer.send(msg);
System.out.println("Message send result: " + sendResult);
} finally {
// 4. Shut down producer
producer.shutdown();
}
}
}

5.2 Consumer with Custom TraceTopic

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
public class CustomTraceConsumer {
public static void main(String[] args) throws Exception {
// 1. Create consumer, enable tracing, specify custom trace topic
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("OrderConsumerGroup", true, "MyApp_TraceTopic");
consumer.setNamesrvAddr("localhost:9876");
// 2. Subscribe business topic
consumer.subscribe("OrderTopic", "PayOrder");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
// 3. Register listener
consumer.registerMessageListener((MessageListenerConcurrently) (msgs, context) -> {
System.out.println("Received order message: " + new String(msgs.get(0).getBody()));
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
});
consumer.start();
System.out.println("Consumer started, waiting for order messages...");
}
}

5.3 Trace Data Verification

Ensure the custom trace Topic exists (create manually if auto‑create is disabled): sh mqadmin updateTopic -n localhost:9876 -t MyApp_TraceTopic -c DefaultCluster

Run the producer to send a test message to OrderTopic ; trace data will be written to MyApp_TraceTopic .

Query the trace using RocketMQ console or CLI: sh mqadmin QueryMsgTraceById -n localhost:9876 -i "0A1234567890ABCDEF" # replace with actual MsgID