Monitoring GPU Metrics with DCGM Exporter and Prometheus

Why simple nvidia-smi is not enough

When GPUs are used for training or inference, nvidia-smi only shows a snapshot and cannot answer questions about memory growth over the past half‑hour, Xid errors, throttling, or which Kubernetes Pods are consuming GPUs. DCGM Exporter converts NVIDIA Data Center GPU Manager (DCGM) fields into Prometheus metrics so that GPU state can be continuously scraped, queried and alerted.

Monitoring chain

The chain consists of GPU driver + DCGM, the dcgm‑exporter running on each node, Prometheus scraping, and alerting/visualisation. Any mis‑configuration at a layer can result in “no GPU metrics”. Troubleshooting must verify each layer rather than reinstalling drivers when Grafana shows empty panels.

What can be monitored

DCGM_FI_DEV_GPU_UTIL

: GPU utilization percentage – detect long idle or sustained full load. DCGM_FI_DEV_MEM_COPY_UTIL: Memory copy engine utilization – shows data movement activity. DCGM_FI_DEV_FB_USED: Framebuffer memory used (MiB) – monitor memory trends, leaks or capacity limits. DCGM_FI_DEV_FB_FREE: Framebuffer memory free (MiB) – remaining memory. DCGM_FI_DEV_GPU_TEMP: GPU temperature (°C) – heat and throttling risk. DCGM_FI_DEV_POWER_USAGE: Power draw (W) – power fluctuations and limits. DCGM_FI_DEV_SM_CLOCK: SM clock frequency (MHz) – possible throttling. DCGM_FI_DEV_XID_ERRORS: Last Xid error code (0 means no error) – hardware or driver anomalies.

When DCGM profiling is enabled, additional fields such as SM activity, Tensor Core activity and memory bandwidth are exposed. The exporter does not replace a framework profiler and does not automatically pinpoint model‑level bottlenecks.

Pre‑deployment checks on the host

Run the following read‑only script to verify driver, GPU enumeration and device nodes. If nvidia-smi fails, fix driver or hardware before deploying the exporter because it relies on a healthy NVIDIA driver.

#!/usr/bin/env bash
set -euo pipefail

nvidia-smi
nvidia-smi --query-gpu=index,uuid,name,driver_version,memory.total --format=csv,noheader
ls -l /dev/nvidia* 2>/dev/null || true
uname -r

# Verify NVIDIA Container Toolkit
docker run --rm --gpus all <CUDA_BASE_IMAGE> nvidia-smi --query-gpu=index,uuid,name --format=csv,noheader

Ensure the chosen <CUDA_BASE_IMAGE> matches the driver version. If the container fails while the host succeeds, investigate Docker runtime, NVIDIA Container Toolkit and device permissions before touching Prometheus.

Running dcgm‑exporter with Docker

Never use the latest tag in production; pin a specific image digest. The exporter needs --gpus all to expose GPUs and --cap-add SYS_ADMIN for DCGM; avoid --privileged.

IMAGE="<DCGM_EXPORTER_IMAGE>:<TAG>"

docker image inspect "$IMAGE" --format 'Id={{.Id}} Digests={{json .RepoDigests}}'

docker run --rm "$IMAGE" --help

docker run -d \
  --name dcgm-exporter \
  --restart unless-stopped \
  --gpus all \
  --cap-add SYS_ADMIN \
  -p 127.0.0.1:9400:9400 \
  <DCGM_EXPORTER_IMAGE>:<TAG>

Binding to 127.0.0.1 is suitable when Prometheus runs on the same host; for cross‑host scraping bind to a controlled internal address and update the Prometheus target accordingly.

Compose deployment

A typical docker‑compose.yml snippet:

services:
  dcgm-exporter:
    image: "<DCGM_EXPORTER_IMAGE>:<TAG>"
    container_name: dcgm-exporter
    restart: unless-stopped
    cap_add:
      - SYS_ADMIN
    ports:
      - "<NODE_IP>:9400:9400"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD-SHELL","wget -qO- http://127.0.0.1:9400/metrics >/dev/null || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s
    logging:
      driver: json-file
      options:
        max-size: 50m
        max-file: "3"

Replace placeholders with the actual image, tag and node IP. Verify the rendered configuration before applying.

Prometheus scrape configuration

For a small set of bare‑metal GPU nodes, use a static file‑based target list:

# dcgm-exporter.yaml
- targets:
  - "10.20.10.11:9400"
  - "10.20.10.12:9400"
  labels:
    env: "production"
    cluster: "gpu-baremetal"

Add the file to prometheus.yml:

scrape_configs:
  - job_name: dcgm-exporter-baremetal
    scrape_interval: 15s
    scrape_timeout: 10s
    file_sd_configs:
      - files:
        - /etc/prometheus/targets/dcgm-exporter.yaml
    refresh_interval: 1m

Validate the configuration with promtool check config and reload only after a successful syntax check.

Kubernetes deployment

When the NVIDIA GPU Operator is present, check whether it already manages a dcgm‑exporter DaemonSet to avoid duplicate exporters on the same node.

# List GPU nodes and existing workloads
kubectl get nodes -n <MONITOR_NS> -l nvidia.com/gpu.present=true -o custom-columns=NAME:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu

# Inspect DaemonSet, Pods and Service
kubectl get daemonset,pod,service -n <MONITOR_NS> -o wide | grep -Ei 'dcgm|gpu' || true

When using the Helm chart, render the values first, fix the image tag, service type, ServiceMonitor and node selector, then install with helm upgrade --install and the --atomic flag for automatic rollback on failure.

Verification steps

Check that the Service selector matches the Pods and that the endpoint list contains port 9400.

Use curl http://<EXPORTER_IP>:9400/metrics on the Prometheus host to confirm metric text, including # HELP and # TYPE lines.

If the exporter returns metrics but Prometheus shows the target down, examine network policies, DNS resolution and the ServiceMonitor spec.endpoints[].port name.

Validate that GPU UUIDs are mapped to Pod/namespace labels via the kubelet /var/lib/kubelet/pod-resources socket.

Custom collectors

dcgm‑exporter reads a CSV file that lists field IDs, Prometheus types and help strings. To add or replace fields, locate the collector file inside the container (paths vary by image) and mount a read‑only custom CSV.

# List collector files inside the container
docker exec dcgm-exporter sh -c '
  tr "\0" " " </proc/1/cmdline; echo
  find /etc /opt -type f \( -name "*counters*.csv" -o -name "*collectors*.csv" \) 2>/dev/null
'

# Example custom CSV (first three columns)
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %).
DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %).
DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB).
DCGM_FI_DEV_GPU_TEMP, gauge, GPU temperature (in C).
DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (in W).
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last XID error encountered.

Set the environment variable DCGM_EXPORTER_COLLECTORS to point to the custom file, but only after confirming the exporter version supports the variable.

PromQL queries

# Exporter health
up{job=~".*dcgm.*"}

# 5‑minute average GPU utilization
avg_over_time(DCGM_FI_DEV_GPU_UTIL[5m])

# Memory usage percentage
100 * DCGM_FI_DEV_FB_USED / clamp_min(DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE, 1)

# Highest temperature in last 5 minutes
max_over_time(DCGM_FI_DEV_GPU_TEMP[5m])

# Detect non‑zero Xid in the last 10 minutes
max_over_time(DCGM_FI_DEV_XID_ERRORS[10m]) > 0

# Average utilization per Kubernetes namespace
avg by (namespace) (DCGM_FI_DEV_GPU_UTIL{namespace!=""})

Alerting rules (example)

groups:
- name: gpu-dcgm
  rules:
  - alert: DCGMExporterDown
    expr: up{job=~".*dcgm.*"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "DCGM Exporter target is down"

  - alert: GPUTemperatureHigh
    expr: DCGM_FI_DEV_GPU_TEMP > 85
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "GPU temperature remains above the reviewed threshold"

  - alert: GPUMemoryNearlyFull
    expr: 100 * DCGM_FI_DEV_FB_USED / clamp_min(DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE, 1) > 95
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: "GPU framebuffer memory usage is high"

  - alert: GPUXidDetected
    expr: max_over_time(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "A non‑zero NVIDIA Xid was reported"

Validate rule syntax with promtool check rules before reloading Prometheus.

Troubleshooting checklist

Exporter container fails to start – inspect exit code, capabilities and logs.

Kubernetes Pod stuck in Pending or CrashLoopBackOff – describe the pod, check node selector, taints, resource limits. /metrics works locally but Prometheus target is down – test from the Prometheus host or a temporary debug pod.

Metric gaps – use absent_over_time(DCGM_FI_DEV_GPU_UTIL[5m]) to find missing series and correlate with pod restarts.

Xid alerts – locate the GPU UUID in the alert, then query nvidia-smi and kernel logs on the node for the corresponding error.

Upgrade and rollback

Before upgrading, back up the current Compose file, custom CSV and the list of exposed metric names. Record the image digest with docker image inspect. After upgrading, verify that the DaemonSet, Service endpoints, Prometheus targets and alert rules still function. If not, roll back with helm rollback (for Helm) or redeploy the previous image (for bare‑metal).

Daily health‑check script

#!/usr/bin/env bash
set -euo pipefail

EXPORTER_URL="${EXPORTER_URL:-http://127.0.0.1:9400/metrics}"
PROM_URL="${PROM_URL:-http://127.0.0.1:9090}"
CONTAINER="${CONTAINER:-dcgm-exporter}"

# Verify container is running
if [[ "$(docker inspect -f '{{.State.Running}}' "$CONTAINER")" != "true" ]]; then
  echo "CRITICAL: $CONTAINER is not running" >&2
  exit 2
fi

metrics=$(curl -fsS --max-time 10 "$EXPORTER_URL")
for name in DCGM_FI_DEV_GPU_UTIL DCGM_FI_DEV_FB_USED DCGM_FI_DEV_GPU_TEMP; do
  grep -q "^${name}{" <<<"$metrics" || { echo "CRITICAL: missing metric $name" >&2; exit 2; }
done

# Verify Prometheus can see the exporter
curl -fsS "$PROM_URL/api/v1/query?query=up%7Bjob%3D~%22.*dcgm.*%22%7D" |
  jq -e '.status == "success" and (.data.result | length > 0)' >/dev/null

echo "OK: exporter and Prometheus query succeeded"

Acceptance checklist

Host and container can access the GPU.

Exporter image tag is fixed (no latest).

Port 9400 is only reachable from the trusted network. /metrics contains expected GPU UUIDs and the fields listed above.

Prometheus target shows up = 1.

Kubernetes Service has an endpoint and ServiceMonitor port name matches the Service.

Pod labels (namespace, pod, container) are correctly propagated.

PromQL queries use real metric names and labels.

Alert thresholds are reviewed against device specifications and historical baselines.

Upgrade and rollback procedures are documented and tested.