How to Expose and Collect Metrics with OpenTelemetry and Prometheus in Cloud‑Native Java Apps

Background

Metrics are fundamental to cloud‑native observability. Prometheus introduced the metric model, and OpenTelemetry provides a vendor‑neutral API that can export to Prometheus, OTLP, or other backends.

Core components

MeterProvider and Meter

OpenTelemetry uses a MeterProvider to obtain a Meter. The Meter creates instrument objects (Counter, UpDownCounter, Gauge, Histogram). Example (Pulsar client):

public class InstrumentProvider {
    private final Meter meter;

    public InstrumentProvider(OpenTelemetry otel) {
        if (otel == null) {
            // Metrics are disabled unless the OTel Java agent is configured.
            otel = GlobalOpenTelemetry.get();
        }
        this.meter = otel.getMeterProvider()
            .meterBuilder("org.apache.pulsar.client")
            .setInstrumentationVersion(PulsarVersion.getVersion())
            .build();
    }

    public LongCounterBuilder counterBuilder(String name, String description, String unit) {
        return meter.counterBuilder(name)
            .setDescription(description)
            .setUnit(unit);
    }
}

Exporters

Exporters send metric data to a backend. The most common are:

OTLP exporter – uses the OpenTelemetry Protocol. Enable with -Dotel.metrics.exporter=otlp (default).

Console exporter – prints metrics to STDOUT. Enable with -Dotel.metrics.exporter=console.

Prometheus exporter – exposes an HTTP endpoint that Prometheus can scrape. Enable with -Dotel.metrics.exporter=prometheus and configure the port with -Dotel.exporter.prometheus.port=<port>.

Instrument types

Counter – monotonic increasing value (e.g., total requests).

UpDownCounter – can increase or decrease.

Gauge – records instantaneous values (e.g., memory usage).

Histogram – records distribution of values such as latency.

Each instrument requires a name (mandatory), kind (mandatory), and may include an optional unit and description.

Creating instruments – Pulsar examples

Counter for total messages received:

LongCounter messageInCounter = meter
    .counterBuilder("pulsar.message.in.total")
    .setUnit("{message}")
    .setDescription("Total number of messages received for a topic")
    .buildObserver();

UpDownCounter for subscription count:

LongUpDownCounter subscriptionCounter = meter
    .upDownCounterBuilder("pulsar.subscription.count")
    .setUnit("{subscription}")
    .setDescription("Current number of Pulsar subscriptions")
    .buildObserver();

Histogram for producer latency (seconds) with explicit bucket boundaries:

List<Double> latencyBuckets = List.of(
    0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05,
    0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0);

DoubleHistogram latencyHistogram = meter
    .histogramBuilder("pulsar.producer.latency")
    .setDescription("Publish latency experienced by the client, including batching")
    .setUnit("s")
    .setExplicitBucketBoundariesAdvice(latencyBuckets)
    .build();

Gauge for backlog age (seconds) using a callback:

meter.gaugeBuilder("pulsar.backlog.age")
    .ofLongs()
    .setUnit("s")
    .setDescription("Age of the oldest unacknowledged message")
    .buildWithCallback(observable -> {
        for (Producer producer : CollectionHelper.PRODUCER_COLLECTION.list()) {
            ProducerStats stats = producer.getStats();
            String topic = producer.getTopic();
            if (topic.endsWith(RetryMessageUtil.RETRY_GROUP_TOPIC_SUFFIX)) {
                continue;
            }
            observable.record(
                stats.getNumMsgsSent(),
                Attributes.of(
                    AttributeKey.stringKey("producer.name"), producer.getProducerName(),
                    AttributeKey.stringKey("topic"), topic));
        }
    });

Running with the Prometheus exporter

Start the Java application with the OpenTelemetry Java agent and the Prometheus exporter:

java -javaagent:opentelemetry-javaagent.jar \
    -Dotel.javaagent.extensions=ext.jar \
    -Dotel.metrics.exporter=prometheus \
    -Dotel.exporter.prometheus.port=18180 \
    -jar myapp.jar

Metrics are available at http://127.0.0.1:18180/metrics. They can be scraped directly by Prometheus or forwarded to an OpenTelemetry Collector.

Collector configuration (YAML)

exporters:
  otlphttp:
    metrics_endpoint: http://prometheus:8480/insert/0/opentelemetry/api/v1/push
  debug: {}

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [otlphttp, debug]

Dependencies

When using OpenTelemetry instead of the Prometheus client, add the following Maven coordinates (example versions):

compileOnly 'io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.34.1'
compileOnly 'io.opentelemetry.instrumentation:opentelemetry-instrumentation-api:1.32.0'

Best practices

Follow the OpenTelemetry semantic conventions for metric naming (dot‑separated hierarchy).

Prefer the OTLP exporter for production deployments; the Prometheus exporter is useful for compatibility with existing Prometheus setups.

Use callbacks for gauge‑type data that should be sampled at a regular interval (default ~30 s).

Correlate metrics with traces via exemplars when detailed troubleshooting is required.

References

https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/metrics/InstrumentProvider.java

https://opentelemetry.io/docs/specs/semconv/general/metrics/

https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exemplars