How RocketMQ 5.0 Enables Distributed End‑to‑End Tracing with OpenTelemetry

Background

In large‑scale distributed systems, locating the root cause of failures across many services is difficult. End‑to‑end tracing records the full request path and performance data, improving reliability and maintainability.

OpenTelemetry as the de‑facto standard

OpenTelemetry (the successor of OpenTracing and OpenCensus) defines a unified tracing specification. It adopts the span concept introduced by Google’s Dapper paper (2007) and adds semantic conventions for messaging systems.

Messaging span conventions for RocketMQ 5.0

send : starts when a message is sent and ends on success or failure; each retry creates a new span.

receive : represents the long‑polling receive lifecycle in a consumer (disabled by default).

process : covers the processing of a message inside MessageListener of a PushConsumer.

If the receive span is disabled, the process span becomes a child of the send span. When enabled, process is a child of receive and links back to send.

Instrumentation approaches

Automatic instrumentation : attach the OpenTelemetry Java agent at runtime (e.g., -javaagent:/path/to/opentelemetry-javaagent.jar). No code changes are required.

Manual instrumentation : write code to create spans and export them, giving fine‑grained control.

RocketMQ 5.0’s client tracing is built on automatic instrumentation; the Java agent automatically instruments the RocketMQ client library.

Quick start

Obtain the RocketMQ 5.0 Java client from the rocketmq-clients repository (https://github.com/apache/rocketmq-clients).

Download the latest OpenTelemetry Java agent (https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar).

Start the application with the agent:

java -javaagent:/path/to/opentelemetry-javaagent.jar -jar your-app.jar

Optionally set the collector endpoint via the environment variable OTEL_EXPORTER_OTLP_ENDPOINT.

To enable the receive span, add the JVM option:

-Dotel.instrumentation.messaging.experimental.receive-telemetry.enabled=true

Example scenario

The rocketmq-opentelemetry demo launches three processes:

A gRPC client calls a gRPC server.

The server sends a message using a RocketMQ 5.0 producer.

The producer’s PushConsumer receives the message, invokes an HttpClient request to an external site, and processes the response.

With the default configuration (receive span disabled), the trace contains seven spans in chronological order:

ExampleUpstreamSpan (custom upstream business span).

gRPC client request span.

gRPC server response span.

RocketMQ send span.

RocketMQ process span.

HTTP request span.

ExampleDownstreamSpan (custom downstream business span).

Exporting traces

Alibaba Cloud SLS (Log Service) Trace

Create a Trace service instance in SLS.

Configure the endpoint, project, and instance name according to the SLS integration guide.

Run the instrumented application; traces appear in the SLS console and can be queried with SQL‑like syntax.

Alibaba Cloud ARMS (Application Real‑Time Monitoring)

In the ARMS console, enable OpenTelemetry for Java applications.

Apply the generated startup parameters (e.g., the -javaagent flag) to the Java process.

Traces become visible in the ARMS Trace Explorer, where span attributes, resources, events, and temporal relationships can be inspected.

Messaging attributes (common)

messaging.message.id

: unique identifier of the message. messaging.destination: destination name (queue or topic). messaging.operation: operation type (send, receive, acknowledge, etc.).

RocketMQ‑specific attributes (selected)

messaging.rocketmq.client_group

(string): producer/consumer load‑balancing group (effective for consumers in 5.0). messaging.rocketmq.client_id (string): unique client identifier. messaging.rocketmq.message.delivery_timestamp (int): scheduled delivery time for delayed messages (5.0 only). messaging.rocketmq.message.type (string): message type (normal/fifo/delay/transaction, 5.0 only). messaging.rocketmq.message.tag (string): message tag. messaging.rocketmq.message.keys (string[]): message keys, can be multiple.

Trends and future directions

Observability is becoming a core DevOps pillar. Emerging technologies such as eBPF and Service Mesh influence tracing implementations. RocketMQ 5.0’s proxy already incorporates Service Mesh concepts, and future work aims to merge client‑side and server‑side traces and correlate trace data with metrics via exemplars.

References

Dapper paper: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36356.pdf

OpenTelemetry messaging semantic conventions: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/messaging.md

RocketMQ client repository: https://github.com/apache/rocketmq-clients

OpenTelemetry Java instrumentation for RocketMQ 5.0: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/rocketmq/rocketmq-client/rocketmq-client-5.0

RocketMQ OpenTelemetry demo: https://github.com/aaron-ai/rocketmq-opentelemetry

SLS Trace integration guide: https://help.aliyun.com/document_detail/208894.html

ARMS OpenTelemetry integration guide: https://help.aliyun.com/document_detail/407604.html