Developing an OpenTelemetry Extension for Pulsar Java Client Metrics

This article describes how to create a custom OpenTelemetry extension that exposes Pulsar Java client metrics. It walks through the migration from SkyWalking to OpenTelemetry, the overall development workflow, and the necessary code components.

Development workflow

The extension is built using the same byte‑code enhancement library (ByteBuddy) as SkyWalking, but with OpenTelemetry APIs. The steps include creating a Java project, writing the core javaagent class, defining instrumentations, implementing advice methods, registering custom metrics, and packaging the extension.

Creating the project

A Gradle project is used (Maven is also possible). Below is a simplified build.gradle configuration:

plugins {
id 'java'
id "com.github.johnrengelman.shadow" version "8.1.1"
id "com.diffplug.spotless" version "6.24.0"
}
group = 'com.xx.otel.extensions'
version = '1.0.0'
ext {
versions = [
opentelemetrySdk           : "1.34.1",
opentelemetryJavaagent     : "2.1.0-SNAPSHOT",
opentelemetryJavaagentAlpha: "2.1.0-alpha-SNAPSHOT",
junit                     : "5.10.1"
]
deps = [
autoservice: dependencies.create(group: 'com.google.auto.service', name: 'auto-service', version: '1.1.1')
]
}
repositories {
mavenLocal()
maven { url "https://maven.aliyun.com/repository/public" }
mavenCentral()
}
configurations { otel }
dependencies {
implementation(platform("io.opentelemetry:opentelemetry-bom:${versions.opentelemetrySdk}"))
compileOnly 'io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.34.1'
compileOnly 'io.opentelemetry.instrumentation:opentelemetry-instrumentation-api:1.32.0'
compileOnly 'io.opentelemetry.javaagent:opentelemetry-javaagent-extension-api:1.32.0-alpha'
compileOnly deps.autoservice
annotationProcessor deps.autoservice
compileOnly 'org.apache.pulsar:pulsar-client:2.8.0'
}
test { useJUnitPlatform() }

The core javaagent class registers the extension name and uses @AutoService to generate the SPI file under META-INF/services/ :

@AutoService(InstrumentationModule.class) 
public class PulsarInstrumentationModule extends InstrumentationModule { 
    public PulsarInstrumentationModule() { 
        super("pulsar-client-metrics", "pulsar-client-metrics-2.8.0"); 
    } 
}

Creating Instrumentation

Each instrumentation defines which class and method to intercept. Example for intercepting ProducerBuilderImpl.createAsync :

public class ProducerCreateImplInstrumentation implements TypeInstrumentation { 
    @Override 
    public ElementMatcher
typeMatcher() { 
        return named("org.apache.pulsar.client.impl.ProducerBuilderImpl"); 
    } 
    @Override 
    public void transform(TypeTransformer transformer) { 
        transformer.applyAdviceToMethod(
            isMethod().and(named("createAsync")), 
            ProducerCreateImplInstrumentation.class.getName() + "$ProducerCreateImplConstructorAdvice"); 
    } 
}

The advice class contains two methods annotated with @Advice.OnMethodEnter and @Advice.OnMethodExit :

public static class ProducerCreateImplConstructorAdvice { 
    @Advice.OnMethodEnter(suppress = Throwable.class) 
    public static void onEnter() { 
        MetricsRegistration.registerProducer(); 
    } 
    @Advice.OnMethodExit(suppress = Throwable.class) 
    public static void after(@Advice.Return CompletableFuture
completableFuture) { 
        try { 
            Producer producer = completableFuture.get(); 
            CollectionHelper.PRODUCER_COLLECTION.addObject(producer); 
        } catch (Throwable e) { 
            System.err.println(e.getMessage()); 
        } 
    } 
}

Two important advice annotations are @Advice.OnMethodEnter (executed before the target method) and @Advice.OnMethodExit (executed after the target method). Additional annotations such as @Advice.Return and @Advice.This can retrieve the return value or the instance.

Custom metrics

The extension uses the OpenTelemetry metrics API to expose custom gauges. Example registration:

public static void registerObservers() { 
    Meter meter = MetricsRegistration.getMeter(); 
    meter.gaugeBuilder("pulsar_producer_num_msg_send") 
        .setDescription("The number of messages published in the last interval") 
        .ofLongs() 
        .buildWithCallback(r -> recordProducerMetrics(r, ProducerStats::getNumMsgsSent)); 
}

private static void recordProducerMetrics(ObservableLongMeasurement observableLongMeasurement, Function
getter) { 
    for (Producer producer : CollectionHelper.PRODUCER_COLLECTION.list()) { 
        ProducerStats stats = producer.getStats(); 
        String topic = producer.getTopic(); 
        if (topic.endsWith(RetryMessageUtil.RETRY_GROUP_TOPIC_SUFFIX)) { 
            continue; 
        } 
        observableLongMeasurement.record(getter.apply(stats), 
            Attributes.of(PRODUCER_NAME, producer.getProducerName(), TOPIC, topic)); 
    } 
}

Running the application with the extension is as simple as adding the -javaagent flag:

java -javaagent:opentelemetry-javaagent.jar \
    -Dotel.javaagent.extensions=ext.jar \
    -jar myapp.jar

Packaging can be done with ./gradlew build to produce a JAR in build/libs/ . The extension can also be merged into the original opentelemetry-javaagent.jar using a custom Gradle task ( extendedAgent ) that copies the extension JAR into the extensions directory inside the agent.

Pitfalls

NoClassDefFoundError : Occurs when helper classes are not added to the application class loader. The solution is to override getAdditionalHelperClassNames in the instrumentation module.

Missing logs : By default, advice methods suppress exceptions silently. Adding suppress = Throwable.class and explicit try‑catch blocks makes errors visible.

Debugging metrics : OpenTelemetry supports several exporters. Use -Dotel.metrics.exporter=logging for console output, -Dotel.metrics.exporter=otlp for the default collector, or -Dotel.metrics.exporter=prometheus for Prometheus scraping.

Operator configuration

The OpenTelemetry Operator currently does not expose an extensions field. The article proposes adding the extension JAR to a custom image and setting the environment variable OTEL_EXTENSIONS_DIR so that the operator can pass -Dotel.javaagent.extensions=${java.extensions} to the agent.

Conclusion

The process of building an OpenTelemetry extension for Pulsar is straightforward once the byte‑code instrumentation concepts are understood. The article provides a complete example, common pitfalls, and guidance on packaging and deployment, helping developers quickly add custom observability to their Java applications.