How to Build a Full‑Featured Kubernetes Monitoring Stack with Prometheus & OpenTelemetry

Monitoring System Components

Metric exposure – expose internal metrics that need to be observed.

Metric collection – gather and store the exposed metrics.

Metric visualization – display and analyze data with charts.

Alerting – notify when critical metrics breach defined thresholds.

Kubernetes System Metrics (cAdvisor)

cAdvisor is built‑in to Kubernetes and provides node‑level metrics such as CPU, memory, and filesystem usage. Add a scrape job to Prometheus to collect these metrics:

- job_name: nodeScrape/monitoring/cadvisor-scrape/0
  scrape_interval: 30s
  scrape_timeout: 15s
  scheme: https
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
    - source_labels: [__meta_kubernetes_node_name]
      target_label: node
    - action: replace
      source_labels: [__meta_kubernetes_node_name]
      separator: ;
      target_label: __address__
      regex: (.*)
      replacement: kubernetes.default.svc:443
    - action: replace
      source_labels: [__meta_kubernetes_node_name]
      separator: ;
      target_label: __metrics_path__
      regex: (.+)
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
  kubernetes_sd_configs:
    - role: node

Additional endpoints such as /metrics/resource and /metrics/probes are also exposed. Full metric list: https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md#prometheus-container-metrics

Application Metrics (Java / Spring Boot)

Use the Prometheus Java client libraries to expose JVM metrics. Maven dependencies:

<dependency>
  <groupId>io.prometheus</groupId>
  <artifactId>simpleclient</artifactId>
  <version>0.16.0</version>
</dependency>
<dependency>
  <groupId>io.prometheus</groupId>
  <artifactId>simpleclient_hotspot</artifactId>
  <version>0.16.0</version>
</dependency>

Static scrape configuration (useful for local testing):

scrape_configs:
  - job_name: 'springboot'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:8080']  # Spring Boot ip+port

In Kubernetes the pod IP changes, so use service discovery with annotations:

- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_label_component]
      action: replace
      target_label: job
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

Add the following annotations to the pod template so Prometheus discovers the endpoint:

template:
  metadata:
    annotations:
      prometheus.io/path: /metrics
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"

When the target status in the Prometheus UI shows UP , scraping is successful.

OpenTelemetry Integration

Instead of Prometheus pulling metrics, the application pushes logs, traces, and metrics to an OpenTelemetry Collector, which then forwards them to a backend such as Prometheus.

Attach the OpenTelemetry Java agent to the JVM:

java -javaagent:opentelemetry-javaagent-2.4.0-SNAPSHOT.jar \
  -Dotel.traces.exporter=otlp \
  -Dotel.metrics.exporter=otlp \
  -Dotel.logs.exporter=none \
  -Dotel.service.name=java-demo \
  -Dotel.exporter.otlp.protocol=grpc \
  -Dotel.propagators=tracecontext,baggage \
  -Dotel.exporter.otlp.endpoint=http://127.0.0.1:5317 \
  -jar target/demo-0.0.1-SNAPSHOT.jar

Configure the OpenTelemetry Collector to forward metrics to Prometheus:

exporters:
  otlphttp/prometheus:
    endpoint: http://prometheus:9292/api/v1/otlp
    tls:
      insecure: true

Installation & Alternatives

Install Prometheus using the official operator or Helm chart (GitHub repository: https://github.com/prometheus-operator/kube-prometheus). For a lighter‑weight, Prometheus‑compatible store, consider VictoriaMetrics.

Additional reference links:

Kubernetes system‑metrics documentation: https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/

cAdvisor Prometheus integration guide: https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md

Prometheus configuration reference: https://prometheus.io/docs/prometheus/latest/configuration/configuration/