DCGM Exporter Mastery: From Sampling Internals to Production Alerting for GPU Monitoring
This comprehensive guide covers DCGM Exporter deployment modes, metric semantics, sampling internals, version-specific breaking changes, PromQL pitfalls, and three real-world failure cases with root-cause analysis and alerting rules for production GPU monitoring.
Introduction: The First-Principle Problem of GPU Monitoring Is Sampling, Not Metric Names
Most teams adopt GPU monitoring by installing the GPU Operator, importing a community Grafana dashboard, and setting three alerts (temperature, memory, utilization), then consider monitoring complete. However, when real issues arise, this setup gives three false signals:
All metrics green but training stuck for 40 minutes — GPU utilization 91%, memory 78%, temperature 72°C, yet the job hangs in NCCL collective with SMs spinning; "busy" does not equal "computing".
HBM degradation invisible — Only DCGM_FI_DEV_ECC_SBE_VOL_TOTAL rises from 3 to 5, but this metric is not enabled by default.
Missing GPU silently drops from metrics — One of eight GPUs marked Unused by kernel; its metric series disappears from /metrics, yet sum(util)/count(util) still shows 92% average, hiding the loss.
These false signals point to one truth: DCGM Exporter is not a mere metric exporter; it is a telemetry pipeline with sampling semantics, a permission model, and field capability probing. Without understanding the pipeline's three stages, the numbers are just digits, not evidence.
1.1 Complete Data Path from Silicon to Prometheus
+--------------- GPU Node (8x A100-SXM4-80G) ---------------+ | [GPU 0..7] NVML/DCGM fields: util / FB / Xid / PCIe | | v | +------------------ DCGM Process ----------------------+ | | Mode A embedded: exporter runs host engine in-process | | | Thread calls libdcgm directly, no separate process | | Mode B standalone: nv-hostengine runs as daemon | | | exporter connects via 127.0.0.1:5555 or unix socket | | Collection schedule = field group x sampling interval | | | DCGM side min 100ms, exporter -c default 30000ms | +---------------------------------------------------+ | | Latest sampled value (sample cache, not streaming) | v +---------------- dcgm-exporter --------------------+ | | :9400/metrics HTTP text format | | CSV-driven: field -> metric name / type / HELP | | K8s mapping: query kubelet pod-resources for pod labels | +---------------------------------------------------+ +---------------------------- | ----------------------------+ | ServiceMonitor scrape 15s v +------------------- Prometheus -------------------+ | Series ID = metric + {gpu, UUID, pci_bus_id, | | modelName, pod, namespace} | | TSDB stores samples, not "state" | +---------------------------- | --------------------+ v +------------------- Alertmanager -----------------+ | group_wait / group_interval / repeat_interval | +--------------------------------------------------+Three engineering facts to remember:
Middle layer is a "sample cache", not a "live table read". Each /metrics request returns the latest DCGM field watch result; -c controls watch refresh rate. If scrape interval (15s) and collection interval (30s) misalign, you get the same value twice, making rate() produce zero or double slopes.
Sampling intervals come in three tiers, determined by the field itself. DCGM_FI_DEV_GPU_UTIL is 1Hz (device-level field group); DCGM_FI_DEV_XID_ERRORS and ECC counters are 100ms state fields; DCGM_FI_PROF_* (Datacenter Profiling) use 10ms→100ms aggregation windows via SM/HBM hardware counters, highest overhead. Different tiers yield different sample counts in a [1m] window, affecting rate stability.
Sampling interval and jitter are bound together. -c 30000 means a 30-second sliding window; each of the 8 GPUs reads within that window but not synchronously — reading GPU 1 and GPU 8 may differ by tens of milliseconds. For 8-GPU tensor-parallel training, averaging masks the skew; to find the straggler you must use max by(gpu), not avg.
1.2 Core Architectural Contradiction: Three Question Layers, Only One Directly Answered
GPU operational alerts decompose into three layers, but DCGM metric semantics only natively answer one:
Hardware Health — "Can the card still be used?" — Metrics: Xid, ECC, retired pages, row remap. Semantics direct: non-zero means issue.
Resource Usage — "How much of the card is occupied?" — Metrics: FB used/free, *_UTIL_RATIO. Semantics direct: look at ratios.
Effective Compute — "Is the card actually doing useful work?" — Metrics: DCGM_FI_PROF_*, NVLink throughput. Semantics NOT direct — requires composite inference.
The third layer is where most teams fail. DCGM_FI_DEV_GPU_UTIL (new spec DCGM_FI_DEV_GPU_UTIL_RATIO) measures "proportion of sampling cycles where at least one warp resides on SM"; its definition lacks "this is useful compute". This causes four typical misjudgments:
Scenario A: TP=8 sync training, GPU 3 NVLink slowdown util: 97 98 96 99 98 97 98 96 -> all green sm_act: 0.91 0.92 0.90 0.31 ... -> GPU 3 SM active only 1/3 Conclusion: util shows waiting on lock, prof reveals truth Scenario B: Inference small batch (max_num_seqs=8) util: 62 63 61 62 -> looks "underutilized" tensor: 0.88 0.89 0.87 -> tensor pipe near saturation Conclusion: low util not waste, batch cannot grow Scenario C: NCCL allreduce SM spinning util: 100 100 100 100 -> full dram: 0.02 0.02 0.02 -> HBM almost idle Conclusion: waiting on network, not computing Scenario D: MIG sliced, single GPU split into 7 util: each slice 100 -> 7 "full" instances Actual: each slice only 1/7 SM resources Conclusion: denominator changed, utilization loses comparabilityEngineering conclusion: *_UTIL only serves as coarse filter for "is resource occupied"; any "compute utilization" judgment must land on DCP fields, accepting their extra overhead and permission cost. This is the starting point for all subsequent configuration and alert design.
1.3 Version Baseline: 2026 Naming Break You Must Handle
DCGM Exporter and DCGM versions are paired (e.g., 4.6.1-4.8.4). Cross-version mixing is officially unsupported. Critical changes in 4.9.0:
Default collection file: dcp-metrics-included.csv → default-counters.csv (old name triggers startup warning).
Utilization unit: percentage 0–100 → ratio 0–1 ( *_RATIO).
Memory unit: MiB → bytes.
Clock unit: MHz → hertz.
Driver version label: DCGM_FI_DRIVER_VERSION → DCGM_FI_SYSTEM_DRIVER_VERSION.
ConfigMap parameter: -m/--configmap-data deprecated, recommend --config-file YAML v2.
This means scraped PromQL and Grafana dashboards copied from the internet will silently compute 100x or 0.01x wrong after 4.9.0. Dividing utilization by 100 turns 0.42 into 0.0042, alerts never fire; dividing memory by 1024 turns bytes into absurd fractions. These "no error but all numbers wrong" problems are far more dangerous than exporter crashes.
Article engineering baseline (all configs, commands, outputs based on this):
OS Rocky Linux 9.4 / kernel 5.14.0-427.el9 GPU NVIDIA A100-SXM4-80GB x 8 (NVSwitch full mesh) Driver 550.127.05 (matches DCGM 4.6.x) DCGM 4.6.1 (dcgmi / nv-hostengine same version) Exporter dcgm-exporter 4.8.4 (default default-counters.csv) Metric nomenclature old naming (DCGM_FI_DEV_GPU_UTIL, percentage / MiB) K8s v1.31, GPU Operator 24.9, DCGM DaemonSet managed Kubelet scrapes every 15s, Prometheus retains 15 daysPart 2: Deployment Postures — Three Modes, Four Hard Constraints
2.1 Three Deployment Shapes and Their Trade-offs
Container embedded — exporter spawns engine in-process. Use case: single-node, validation, CI. Cost: one engine per container, cannot share.
Container + standalone engine — host engine runs as daemon. Use case: production bare-metal. Cost: maintain systemd and port.
K8s DaemonSet — one Pod per node. Use case: standardized clusters. Cost: Pod mapping overhead, permission surface, MIG needs extra handling.
Critical boundary often missed: dcgmi diag, dcgmi dmon, and exporter each starting their own embedded host engine will contend for profiling counters on the same GPU. NVIDIA profiling fields do not allow two host engines to enable the same metric group simultaneously . This is the root cause of "daytime dcgmi dmon works, nighttime exporter drops metrics".
Therefore, correct production bare-metal posture: install DCGM package on host, run nv-hostengine as persistent daemon, exporter connects via --remote-hostengine-info , letting all tools share the same field subscriptions.
2.2 Pre-flight Checks: Prove Cards and Drivers Are Clean First
Seven commands catch 80% of "installed but no data" issues:
# 1. Are all GPUs recognized by driver (count and model) nvidia-smi --query-gpu=index,name,driver_version,serial --format=csv # 2. Any card in Unused/Off (typical leftover state after GPU drop) nvidia-smi -q | grep -A2 -E 'Retired Pages|Unused|Off ' # 3. Kernel-side Xid history journalctl -k --since "-7 days" | grep -cE 'NVRM: Xid' # 4. Can DCGM fields be read directly (without exporter) dcgmi discovery -l # 5. NVLink topology complete (prerequisite for multi-GPU training) nvidia-smi nvlink -s | head -20 # 6. Fabric manager running (required for NVSwitch systems) systemctl is-active nvidia-fabricmanager # 7. Port 9400 already in use ss -lntp | grep 9400If check #2 shows Unused: Yes, do not proceed with exporter — the corresponding GPU series will vanish from /metrics (detailed in 5.1). Try nvidia-smi -r -i <index> to reset; if still Yes, it's hardware or power issue.
2.3 Bare-metal Deployment: Standalone hostengine + Exporter
Step 1: Install DCGM package and start hostengine. Official repo's datacenter-gpu-manager-4-core provides nv-hostengine.
# Verify repo and version dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo dnf clean all && dnf makecache dnf module list nvidia-driver 2>/dev/null | head -5 # Install (version must match driver, build number per compatibility matrix) dnf install -y datacenter-gpu-manager-4-core-4.6.1Step 2: Manage hostengine with systemd, bind only loopback.
# /etc/systemd/system/nv-hostengine.service [Unit] Description=NVIDIA DCGM Host Engine After=network.target nvidia-persistenced.service Wants=nvidia-persistenced.service # Don't infinite restart if GPU not ready; stop after 5 failures for human intervention StartLimitBurst=5 StartLimitIntervalSec=300 [Service] Type=simple # -b bind loopback: NEVER expose 0.0.0.0, 5555 is unauthenticated port ExecStart=/usr/bin/nv-hostengine -b 127.0.0.1 --port 5555 Restart=on-failure RestartSec=5 # Hard constraint: profiling fields require host engine running as root User=root # Align with exporter collection interval to avoid repeated field subscription rebuild LimitNOFILE=65536 OOMScoreAdjust=-500 [Install] WantedBy=multi-user.targetsystemctl daemon-reload systemctl enable --now nv-hostengine.service systemctl is-active nv-hostengine ss -lntp | grep 5555 dcgmi discovery --host 127.0.0.1:5555 -l | head -3Verify: LISTEN address must be 127.0.0.1:5555; 0.0.0.0:5555 means unauthorized telemetry exposure — anyone can read GPU serials, MIG topology, process list.
Step 3: Run exporter as container connecting to remote hostengine. Note --net=host: remote mode requires connecting to host loopback; bridge network cannot reach.
docker run -d --name dcgm-exporter --restart always
--gpus all
--net=host
--cap-add SYS_ADMIN
--cap-drop ALL
-v /etc/dcgm-exporter:/etc/dcgm-exporter:ro
nvcr.io/nvidia/k8s/dcgm-exporter:4.6.1-4.8.4
--remote-hostengine-info 127.0.0.1:5555
--collectors /etc/dcgm-exporter/default-counters.csv
--collect-interval 15000
--address :9400Why --collect-interval 15000 not default 30000? Collection interval must divide scrape interval ; otherwise you read the same sample twice. 15s scrape with 15s collection is simplest; if Prometheus scrapes at 30s, use -c 30000. Never let collection interval exceed scrape interval — pure waste and causes rate() steps.
2.4 K8s Deployment: Correct DaemonSet Pattern
Most use GPU Operator's managed DCGM DaemonSet, but writing a controllable DaemonSet is essential for troubleshooting — Operator-managed objects are immutable and shouldn't be changed.
Key DaemonSet highlights: nodeSelector: nvidia.com/gpu.present: "true" — avoid empty Pods on GPU-less nodes. hostNetwork: true — required for remote hostengine connectivity.
securityContext: capabilities.add: ["SYS_ADMIN"], runAsUser: 0— SYS_ADMIN is hard requirement for profiling fields, not optional optimization. --kubernetes-gpu-id-type=uid — use UID, device-name collides on same model. --kubernetes-pod-label-allowlist-regex=(app|team|workload) — whitelist only three label types to prevent cardinality explosion.
Mount /var/lib/kubelet/pod-resources read-only for K8s mapping.
Accompanying ConfigMap defines default-counters.csv with only 18 production-needed fields (trimmed from default 27), and ServiceMonitor with 15s scrape, 10s timeout, metric relabeling to drop kube_pod_* labels and alias UUID to stable gpu_uuid.
2.5 Four-Dimensional Parameter Engineering Meaning
① Must-change items (wrong if not changed, or unusable): --address (bind internal IP), --remote-hostengine-info (bare-metal mandatory), --collectors (post-4.9.0 must point to default-counters.csv series), nvidia.com/gpu.present nodeSelector, --kubernetes-gpu-id-type=uid.
② Coupled items (individually fine, combined break): collect-interval × scrape_interval: must be integer multiple or equal; else rate() yields 0 or double steps. --collect-interval × [range window]: PromQL rate(DCGM_FI_PROF_PCIE_TX_BYTES[1m]) meaningless on gauge; any integral metric needs window ≥ 4× collection interval. max_num_seqs × memory reservation: inference batch size determines KV Cache peak; FB_USED alert threshold must derive from "model weights + KV peak + 10% headroom", not fixed 90%. kubernetes.enable_pod_labels × label cardinality: enabling pod labels adds pod, namespace, container labels; frequent Deployment rolls change pod name each time, creating new series per rollout. Must use allowlist-regex to constrain. DCGM_EXP_XID_ERRORS_COUNT window_size_in_ms: default window strongly correlates with --collect-interval; window smaller than collection interval loses events.
③ Performance tuning items (calculate by formula, don't copy numbers):
CPU (cores) ≈ field_count × GPU_count × (1 / collection_interval_sec) × 0.004 Memory (MiB) ≈ 32 + field_count × GPU_count × 0.18 TSDB write rate (samples/sec) ≈ series_count / scrape_interval Baseline verification: Default 27 fields × 8 GPUs = 216 series CPU ≈ 27 × 8 × (1/15) × 0.004 = 0.058 cores Memory ≈ 32 + 27 × 8 × 0.18 = 70.9 MiB Write ≈ 216 / 15 = 14.4 samples/sec/node 200-node cluster = 2880 samples/sec, 15-day retention ≈ 3.7B samplesDefault 27 fields impose negligible TSDB pressure. Real Prometheus explosions come from: enabling full DCP profiling (60+ fields), enabling pod labels, and MIG multiplying entities 7×. MIG scenario: 27 × 8 × 7 = 1512 series/node, 200 nodes = 300k active series — that's the scale needing serious planning.
④ Absolute Red Lines (Never Cross): nv-hostengine -b 0.0.0.0: port 5555 unauthenticated; exposure = GPU serials, MIG config, process list to anyone on network.
Drop SYS_ADMIN: all DCGM_FI_PROF_* fields vanish silently; exporter logs only "Host engine is running as non-root" and skips entire group — you'll think metrics are collecting.
Remove DCGM_FI_DEV_XID_ERRORS from CSV: only in-container signal for GPU drop; removal leaves only kernel logs. --kubernetes-enable-pod-labels=true without allowlist: high-cardinality labels (team, version, build-id) multiply GPU series dozens-fold. -c 1000 (1 second): DCGM min period 100ms, but 1s collection spikes exporter CPU linearly with GPU count; 8-GPU node measured CPU from 0.06 to 0.7 cores, profiling fields start showing last value too old.
Part 3: Metric Semantics — 27 Default Fields Interpreted
default-counters.csv27 fields cover "hardware usable", "resource occupied", "compute running". Below each field's real output shape, interpretation thresholds, and misjudgment traps.
3.1 Raw Output Sample
Real /metrics snippet — note label structure and value dimensions; these matter more than metric names.
DCGM_FI_DEV_GPU_UTIL{gpu="3",UUID="GPU-7f2c...",device="nvidia3", modelName="NVIDIA A100-SXM4-80GB",Hostname="node-gpu-07", DCGM_FI_DRIVER_VERSION="550.127.05", pci_bus_id="00000000:3B:00.0"} 96 DCGM_FI_DEV_FB_USED{gpu="3",...} 74123 DCGM_FI_DEV_GPU_TEMP{gpu="3",...} 71 DCGM_FI_DEV_XID_ERRORS{gpu="3",err_code="0"} 0 DCGM_FI_DEV_POWER_USAGE{gpu="3",...} 381.42Four interpretation keys:
gpu="3" is not PCI index; it's DCGM enumeration order , not necessarily equal to nvidia-smi index. Cross-machine comparison must use UUID as primary key — only stable identifier.
XID_ERRORS carries err_code label ; cardinality = "GPU count × seen error codes"; error code change = new series . Using max_over_time for alerts works; increase() gives wrong delta.
FB_USED unit is MiB (old) or bytes (4.9.0+) ; same expression differs by 1,048,576× across versions.
DCGM_FI_DRIVER_VERSION is label type ; produces no sample rows, appears as label on other metrics. count(DCGM_FI_DRIVER_VERSION) returns empty — common newbie "metric missing" misjudgment.
3.2 Hardware Health Layer: Seven Fields Where Non-Zero Demands Attention
DCGM_FI_DEV_XID_ERRORS: last Xid code; threshold: non-zero in fatal set; trap: only looks at current, misses "reset to zero". DCGM_FI_DEV_PCIE_REPLAY_COUNTER: PCIe replay cumulative; threshold: 1h increase > 10; trap: baseline may have dozens. DCGM_FI_DEV_ECC_SBE_VOL_TOTAL: volatile single-bit ECC; threshold: 24h increase > 5; trap: not enabled by default . DCGM_FI_DEV_ECC_DBE_VOL_TOTAL: volatile double-bit ECC; threshold: >0 immediate action; trap: same as above. DCGM_FI_DEV_RETIRED_SBE/DBE: retired page count; threshold: any growth; trap: confused with ECC. DCGM_FI_DEV_ROW_REMAP_FAILURE: row remap failure; threshold: ==1; trap: ignored. DCGM_FI_DEV_NVLINK_CRC_FLIT_*: NVLink CRC; threshold: 24h increase >0; trap: single-GPU systems always 0, meaningless.
Xid is not "error count" but "last error code" — determines alerting. Same series may go 0 → 79 → 0 in 3 minutes; max_over_time(...[5m]) catches it; DCGM_FI_DEV_XID_ERRORS > 0 misses it.
Fatal Xid set (engineering scope, by remediation action):
13/31/43: illegal access/out-of-bounds → app bug, isolate process.
48: double-bit ECC → immediate offline, replacement workflow.
62/63/64: page retirement/row remap → observe, replace if fails.
74: NVLink error → check fabric manager and links.
79: fallen off bus → reset → if persists, replace.
92: high single-bit ECC rate → planned replacement.
94/95: container unrecoverable → restart container, GPU still usable.
119/120: GSP firmware exception → upgrade or reinstall driver.
3.3 Resource Usage Layer: Why FB_USED Never Equals nvidia-smi
Common question: "Why Grafana shows 74 GiB VRAM, nvidia-smi shows 78 GiB?" Because FB_USED only counts framebuffer allocated by memory manager , while nvidia-smi computes used = total - free, including driver reserved area, CUDA context overhead, ECC metadata regions. reserved field separately tracks driver-reserved but unused portion.
Correct formula uses three fields together:
# VRAM usage (old naming, MiB, pre-4.9.0) (DCGM_FI_DEV_FB_USED) / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE + DCGM_FI_DEV_FB_RESERVED) * 100 # 4.9.0+ uses bytes, semantics identical (DCGM_FI_DEV_FB_USED_BYTES) / (DCGM_FI_DEV_FB_CAPACITY_BYTES) * 100Denominator uses used + free + reserved not hardcoded 81920 because A100 80GB actual usable framebuffer under MIG or ECC is not 81920 MiB . Hardcoded value makes MIG usage stuck at ~30%.
3.4 Effective Compute Layer: DCP Trio and Their Cost
DCGM_FI_PROF_*sourced from GPU hardware performance counters; capability probing uses GPU 0 ( if GPU 0 is different model, probe result may not represent whole machine ). Three core fields: DCGM_FI_PROF_SM_ACTIVE: SM cycles with warp resident ratio; typical full load 0.85–0.97; use to judge "SM actually computing". DCGM_FI_PROF_TENSOR_ACTIVE: Tensor pipe active ratio; typical full load 0.6–0.9; use to judge tensor core saturation. DCGM_FI_PROF_DRAM_ACTIVE: HBM interface active cycles ratio; decode phase 0.7+; use to judge memory bottleneck.
Critical pattern: LLM inference two phases have opposite bottlenecks.
Prefill (prompt processing): parallel matrix multiply dense SM_ACTIVE 0.85+ / TENSOR_ACTIVE 0.7+ / DRAM_ACTIVE 0.35 -> compute-bound, scaling GPUs helps Decode (token generation): each token single GEMV, compute nearly idle SM_ACTIVE 0.95+ / TENSOR_ACTIVE 0.08 / DRAM_ACTIVE 0.78 -> memory-bound, scaling GPUs useless, only increase batchThis explains why same GPU, same VRAM, Decode phase GPU_UTIL hits 98% but actual token/s only 15% of nominal. If alerts only watch GPU_UTIL, you see all green while service severely degraded.
DCP field costs are explicit:
Collection overhead ~10× normal fields; profiling metric groups consume SM registers and counters, 1–3% throughput loss (real cost for training jobs).
Requires SYS_ADMIN; only one host engine per GPU may enable same metric group.
4.9.0+ utilization fields shift to 0–1 ratio; old /100 formulas break entirely.
3.5 Label System: True Determinants of Series Identity
DCGM Exporter labels split into two categories; confusing them yields unjoinable queries:
Entity labels (from DCGM device enumeration) gpu enumeration index, may change on restart UUID GPU-xxxx, stable pci_bus_id 00000000:3B:00.0, physical address, stable modelName model string GPU_I_PROFILE / GPU_I_ID MIG instance (appears only with MIG) GPU_CI_ID MIG compute instance Mapping labels (from Kubernetes / HPC) pod, namespace, container (--use-old-namespace reverts to pod_name / pod_namespace) hpc_job (Slurm environments)Only two stable cross-dimension join paths: UUID and pci_bus_id. pod label disappears on Pod recreation; gpu may reshuffle on driver reload. Any "which task owns this GPU" query should persist UUID and translate via application-side GPU-UUID → task mapping table.
Part 4: Commands and Queries — From One-Liners to Alert Rule Library
4.1 On-Site Forensic One-Liners
Scenario 1: Confirm what exporter actually collects
curl -s http://10.20.31.7:9400/metrics | grep '^DCGM_' | cut -d'{' -f1 | sort | uniq -c | sort -k1nOutput shows per-field GPU count. Profiling fields count 0 means entire group skipped — usually missing SYS_ADMIN. Should be codified in patrol scripts.
Scenario 2: Cross-machine compare same GPU (by UUID)
for h in 10.20.31.{7..12}; do v=$(curl -s --max-time 3 "http://${h}:9400/metrics" | awk '/^DCGM_FI_DEV_GPU_UTIL/ && /gpu="3"/ {print $2; exit}') printf '%s gpu3_util=%s
' "${h}" "${v:-NA}" done--max-time 3mandatory: exporter hangs if host engine unresponsive; script without timeout stalls entire patrol.
Scenario 3: Determine compute vs memory bottleneck
curl -s http://10.20.31.7:9400/metrics | grep -E '^DCGM_FI_PROF_(SM_ACTIVE|TENSOR_ACTIVE|DRAM_ACTIVE)' | awk -F'[{}]' '{split($2,a,","); print $1, a[1], $3}'Interpretation: SM high + TENSOR very low + DRAM high = typical Decode memory-bound . Optimization: increase batch (
max_num_seqs</sub>) or quantize weights, not add GPUs.</p><p><strong>Scenario 4: MIG instance-level localization</strong></p><pre><code>curl -s http://10.20.31.7:9400/metrics | grep '^DCGM_FI_DEV_FB_USED' | grep -o 'GPU_I_PROFILE="[^"]*"|GPU_I_ID="[^"]*"' | paste - -Under MIG, parent GPU aggregate metrics disappear; only instance-level appear. Count lines against expected instance count; missing line = uninitialized instance. Scenario 5: Container vs host metric discrepancy
kubectl exec -n gpu-monitoring ds/dcgm-exporter -- curl -s localhost:9400/metrics | grep -c '^DCGM_' nv=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l) echo "host_gpus=${nv}"Container shows 3 series (only non-profiling), host has 8 GPUs → NVIDIA runtime didn't inject all GPUs . Check NVIDIA_VISIBLE_DEVICES=all and device plugin nvidia.com/gpu.replicas quota.
4.2 Core PromQL: From Coarse Filter to Pinpoint
Queries ordered by decreasing granularity, matching troubleshooting question sequence:
# 1. Cluster GPU avg utilization (overview only, not for alerting) avg(DCGM_FI_DEV_GPU_UTIL) # 2. Per-node GPU avg utilization, find "idle" nodes avg by (Hostname) (DCGM_FI_DEV_GPU_UTIL) # 3. VRAM usage (three-field method, compatible with reserved) (DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE + DCGM_FI_DEV_FB_RESERVED)) * 100 # 4. 8-GPU group "straggler": deviation from group median > 25 points DCGM_FI_DEV_GPU_UTIL - on(Hostname) group_left quantile(0.5, DCGM_FI_DEV_GPU_UTIL) by (Hostname) < -25 # 5. SM active but tensor idle: Decode-bound signal DCGM_FI_PROF_SM_ACTIVE > 0.9 and on(Hostname, UUID) DCGM_FI_PROF_TENSOR_ACTIVE < 0.15 # 6. Prof vs util divergence (former indicates "waiting") DCGM_FI_DEV_GPU_UTIL > 90 and on(Hostname, UUID) DCGM_FI_PROF_SM_ACTIVE < 0.4Query #6 is the article's most important combination: GPU_UTIL high while SM_ACTIVE < 0.4 means SMs have warps but not executing useful instructions — typical NCCL spin, CUDA stream sync, VRAM allocation lock contention. This query has more diagnostic value than any single threshold alert. Why #4 uses median not mean: one GPU drops to 30%, other seven at 97%, mean = 88.6% (looks fine); median = 97%, deviation = 67 points — immediate pinpoint. Quantifies why avg by masks problems.
4.3 K8s-Side Correlation Queries
# Top 5 GPU-util pods (requires --kubernetes=true and pod labels) topk(5, avg by (namespace, pod) (DCGM_FI_DEV_GPU_UTIL) > 50) # Pods holding GPU but zero utilization: resource leak or stuck avg by (namespace, pod) (DCGM_FI_DEV_GPU_UTIL) < 5 and on(namespace, pod) (count by (namespace, pod) (DCGM_FI_DEV_FB_USED) > 0) # Per-namespace GPU VRAM total (by GPU count) sum by (namespace) (DCGM_FI_DEV_FB_USED) / 1024Second query catches "zombie pods holding GPU but doing no work" — typically inference service OOM leaving CUDA context residue; process alive but kernels all failing, GPU_UTIL zero while FB_USED stays high. Manual patrol almost never finds; this alert catches every minute.
4.3 Three Must-Avoid PromQL Traps
Trap 1: rate() on gauge
# Wrong: DCGM_FI_PROF_PCIE_TX_BYTES is already "bytes per second" (gauge) rate(DCGM_FI_PROF_PCIE_TX_BYTES[1m]) # Correct: take value directly or interval average avg_over_time(DCGM_FI_PROF_PCIE_TX_BYTES[1m])rate() on gauge is syntactically legal, Prometheus doesn't error, but returns physically meaningless number . All DCGM_FI_PROF_* bandwidth/utilization fields are gauges; only *_CYCLES_ACTIVE_TOTAL , *_BYTES_TOTAL are counters. Trap 2: XID_ERRORS label drift
# Wrong: err_code change rebuilds series, increase gives wrong delta increase(DCGM_FI_DEV_XID_ERRORS[1h]) # Correct: aggregate across err_code then take max max by (Hostname, gpu) ( max_over_time(DCGM_FI_DEV_XID_ERRORS[1h])) > 0Trap 3: Denominator changed by "disappeared GPU"
# Wrong: dropped GPU node count auto-decreases, utilization "improves" sum(DCGM_FI_DEV_FB_USED) by (Hostname) / sum(DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) by (Hostname) # Correct: verify expected GPU count with separate alert count(DCGM_FI_DEV_FB_USED) by (Hostname) < 8Third query doesn't compute usage; it validates "GPUs still present" . Should be independent alert: count by(Hostname)(DCGM_FI_DEV_FB_USED) < expected_GPU_count — the alert seed for failure 5.1.
Part 5: Three Deep-Water Failure Full Closures
5.1 Case 1: GPU Drop → Metrics Vanish → Alerts Silent Pass
Symptom: 8-GPU training job at hour 6 degrades to 7 GPUs (framework logs NCCL WARN no GPU found ). Grafana shows node GPU avg util rises 89%→92%, VRAM usage 74%→81%, all threshold alerts silent. Step 1: Confirm GPU presence
kubectl exec -n gpu-monitoring ds/dcgm-exporter -- curl -s localhost:9400/metrics | grep -c '^DCGM_FI_DEV_FB_USED' nvidia-smi -L | wc -l nvidia-smi --query-gpu=index,pci.bus_id --format=csvExporter exports 7 series; driver still sees 8 GPUs. Step 2: Identify which GPU and its state
nvidia-smi -q | grep -B4 -A2 'Unused' journalctl -k --since "-8h" | grep -E 'Xid|fallen off|AER' | tail -20Output shows GPU at PCI 0000:3C:00.0 has Unused: Yes and kernel logs Xid 79: GPU has fallen off the bus , then RmInitAdapter failed . Step 3: Root Cause Chain Xid 79 = GPU dropped off PCIe bus → hardware or power layer disconnect , not application-recoverable.
Kernel re-enumerates device ( enabling device) but RmInitAdapter failed → driver cannot initialize GPU into usable context . nvidia-smi still lists it (PCIe device present) but Unused: Yes; DCGM field subscription fails continuously → exporter skips entity on read failure, series vanishes .
Key mechanism: DCGM Exporter handles field read failure by "skip entity for that field", not output 0 or NaN . Prometheus sees no anomaly, only "series count decreased". All avg() , sum()/sum() expressions produce prettier numbers because denominator shrank. Step 4: Fix and Before/After
# Attempt reset (only when no workload on GPU, kills all processes on that GPU) nvidia-smi -r -i 3 # If reset refused (process still using), cordon and drain node kubectl cordon node-gpu-07 kubectl drain node-gpu-07 -n gpu-workload --ignore-daemonsets --delete-emptydir-data # If reset still fails, reboot node for PCIe re-enumeration systemctl rebootPost-reboot verification: Unused: No , exporter series back to 8. Comparison table:
Exporter series: 7 → 8 (only correct signal)
Node avg util: 92% → 89% (mathematical illusion)
VRAM usage: 81% → 74% (denominator shrink illusion)
Xid alert: not fired → not fired ( metric absence caused missed alert )
Must-add alerts (sole value from this incident):
# Rule 1: Fixed expected GPU count (simplest, most reliable) count by (Hostname) (DCGM_FI_DEV_FB_USED) < 8 # Rule 2: Align with node GPU capacity (multi-model clusters) (count by (node) (DCGM_FI_DEV_FB_USED)) < on (node) group_left kube_node_status_capacity{resource="nvidia_com_gpu"}Rollback: if dcgmi diag -r 2 still FAIL after reset, remove node from GPU pool (taint nvidia.com/gpu.present=false ), proceed to replacement. Don't "run on 7 GPUs" — TP=8 jobs timeout on collectives, creating more noise.
5.2 Case 2: Profiling Fields Silently Vanish → Misjudged as "Compute Not Saturated"
Symptom: Inference service scaled up, QPS +30%, but Grafana tensor utilization panel shows "no data". Ops blamed model efficiency, planned framework rewrite. Reality: exporter lost SYS_ADMIN after DaemonSet rollout. Step 1: Confirm field existence
kubectl exec -n gpu-monitoring ds/dcgm-exporter -- curl -s localhost:9400/metrics | grep -c '^DCGM_FI_PROF_' kubectl logs -n gpu-monitoring ds/dcgm-exporter --tail=400 profiling fields. Logs show Error watching fields: Host engine is running as non-root. — 21 fields = 27 - 6 (all DCP). Error level but process doesn't exit — exporter degrades on capability probe failure, no fail-fast. Step 2: Verify Deployment Object
kubectl get ds -n gpu-monitoring dcgm-exporter -o jsonpath='{.spec.template.spec.containers[0].securityContext}' kubectl get ds -n gpu-monitoring dcgm-exporter -o jsonpath='{.spec.template.spec.containers[0].args}'runAsUser: 1000 is root cause. GPU Operator 24.x tightened default securityContext to non-root; DCP capability requirement unchanged. High-frequency regression after Operator upgrade. Step 3: Fix
kubectl set image -n gpu-monitoring ds/dcgm-exporter exporter=nvcr.io/nvidia/k8s/dcgm-exporter:4.6.1-4.8.4 kubectl patch -n gpu-monitoring ds dcgm-exporter --type=merge -p 'spec: template: spec: containers: - name: exporter securityContext: runAsUser: 0 allowPrivilegeEscalation: false capabilities: add: ["SYS_ADMIN"] drop: ["ALL"]' kubectl rollout status -n gpu-monitoring ds/dcgm-exporter --timeout=180sPost-rollout: grep -c '^DCGM_FI_PROF_' returns 6 (6 fields × 8 GPUs = 48 series). Step 4: Before/After Comparison (most telling data)
PROF_ series: 0 → 48 (restored)
Exporter CPU: 0.04 → 0.09 cores (DCP overhead ~2×)
Same QPS TENSOR_ACTIVE: no data → 0.86 ( was saturated all along )
Same QPS SM_ACTIVE: no data → 0.93 (normal)
End-to-end P99 latency: baseline 1.00 → 1.02 (DCP adds ~2% overhead)
Rows 3-4 mean: "compute not saturated" was entirely false signal . Service always saturated; QPS scaling matched linear expectation. Pursuing "framework rewrite" would waste weeks. Rollback: DCP's 1–3% throughput loss is real cost on training clusters. If business cannot accept, rollback by disabling profiling only for training namespaces — two ConfigMaps + two DaemonSets (selected by node label), not global disable.
5.3 Case 3: NVLink Throughput Looks Normal, Training Actually Spinning
Symptom: H100 NVSwitch node TP=8 training TTFT spikes 240ms→410ms, GPU util 97%, VRAM normal, temp normal, no Xid. Monitoring all green, business 70% slower. Step 1: Rule out compute and memory — nvidia-smi shows all GPUs 96-97% util, 62GB VRAM, SM clocks 1980 MHz consistent, no throttle bits. Traditional three alerts say node perfectly healthy. Step 2: Profiling — "computing" vs "waiting"
for g in 0 1 2 3 4 5 6 7; do curl -s localhost:9400/metrics | awk -v G="gpu=\"$g\"" '$0 ~ G { if ($0 ~ /PROF_SM_ACTIVE/) s=$2; if ($0 ~ /PROF_TENSOR_ACTIVE/) t=$2; if ($0 ~ /PROF_DRAM_ACTIVE/) d=$2 } END {printf "gpu%s sm=%.3f tensor=%.3f dram=%.3f
", G, s, t, d}' doneGPU 2: SM_ACTIVE 0.287 , DRAM_ACTIVE 0.742 — SM mostly idle, HBM saturated moving data . Classic collective communication wait fingerprint (SM spins, HBM busy moving comm buffers). Step 3: Verify NVLink Layer
nvidia-smi nvlink --status -i 2 curl -s localhost:9400/metrics | grep '^DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL' | grep 'gpu="2"'Link 3 speed 13.281 GB/s (half of 26.562). H100 single NVLink 53.125 GB/s bidirectional (26.562 unidirectional); 13.281 means link negotiated to x1 width instead of x2 — physical layer downgrade , but link state still "active", no Xid generated. NVLINK_BANDWIDTH_TOTAL 1172.4 vs theoretical 2550 — aggregated value dragged down by degraded link. Step 4: Root Cause and Fix
Software: restart fabric manager (drains node first, interrupts all GPU comms) to renegotiate NVSwitch routing.
Physical: if restart doesn't restore, board/connector issue → hardware repair.
kubectl cordon node-gpu-11 kubectl drain node-gpu-11 -n gpu-workload --ignore-daemonsets --delete-emptydir-data systemctl restart nvidia-fabricmanager sleep 20 nvidia-smi nvlink --status -i 2 | grep 'Link 3'Link 3 restored to 26.562 GB/s. Step 5: Before/After
GPU 2 SM_ACTIVE: 0.287 → 0.908 (+216%)
GPU 2 DRAM_ACTIVE: 0.742 → 0.364 (comm buffer moves gone)
TP=8 TTFT: 410ms → 243ms (-41%) GPU_UTIL: 97% → 96% ( virtually unchanged )
Training throughput: 8.2k → 13.9k tok/s (+70%)
Final comparison is the article's core conclusion: GPU_UTIL changed 97%→96% (near zero) while effective throughput differed 70%. Any GPU monitoring system using GPU_UTIL as primary metric will inevitably miss this entire failure class. Rollback: fabric manager restart risk "links don't come up" → node can't do multi-GPU training (single-GPU inference still works). Rollback action: move node from training pool to single-GPU inference pool, file hardware ticket. Don't repeatedly restart — three consecutive fabric manager failures usually indicate NVSwitch physical fault. Two Must-Add Alerts:
# Alert A: Group SM_ACTIVE straggler (order of magnitude more sensitive than util) (DCGM_FI_PROF_SM_ACTIVE - on(Hostname) group_left quantile(0.5, DCGM_FI_PROF_SM_ACTIVE) by (Hostname)) < -0.3 # Alert B: NVLink rate degradation (current vs node max) DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL < on(Hostname) group_left max by (Hostname) (DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL) * 0.6Part 6: Alert Rule Library and Change Risk Matrix
6.1 Alert Grading Principle: Three Tiers Map to Three Response Actions
P1 (Immediate action, 5-min response) -> GPU unusable / data untrustworthy / service degrading -> Action: isolate node, pull GPU, rollback change P2 (On-duty handling, 1-hour response) -> Significant perf drop but service usable -> Action: locate, reschedule, planned restart P3 (Planned handling, weekly report) -> Degradation trend, capacity watermark -> Action: schedule maintenance windowGrading significance not in color but in deciding "who gets paged at what hour" . Common org problem: all rules set P1, real Xid 79 drowned in hundreds of "VRAM near full" noise.
6.2 Complete Rule Set (PrometheusRule YAML)
Includes groups: gpu.availability (GPUCountMismatch, GPUExporterDown, GPUProfilingFieldsMissing), gpu.health (GPUXidFatal for Xid 79, GPUUncorrectableECC, GPUECCTrend, GPURowRemapFailed, GPUPcieReplayHigh), gpu.performance (GPUSMStarvation, GPUStaggeredInGroup, GPUMemoryPressure, GPUZombiePod, GPUThermalSlowdown, GPUPowerThrottle). Each rule has for duration, severity label, layer label, and actionable annotations. Note: GPUPowerThrottle uses * 0 + 400 hack because DCGM_FI_DEV_BOARD_POWER_LIMIT_REQUESTED_WATTS not enabled by default. Production should add that field to CSV and change rule to POWER_USAGE / POWER_LIMIT > 0.95 . Principle: without limit field, any "near power wall" judgment is guesswork .
6.3 Production Change Risk Matrix
DCGM Exporter changes seem harmless but touch different layers:
Change CSV field set : metric add/remove, Grafana gaps; no outage (SIGHUP reload); diff old/new field list; rollback seconds.
Change -c collection interval : cluster-wide sample distribution; no outage; verify scrape_interval divisibility; rollback seconds.
Upgrade exporter image : metric names/units may change; no outage (rolling); check naming break table per version; rollback 3-5 min.
Enable pod labels : TSDB series explosion; no outage; pre-calculate cardinality, configure allowlist; rollback requires deleting old series.
Switch to remote mode : full metric rebuild; outage risk (engine restart) ; confirm no dcgmi dmon running; rollback 5-10 min.
Change nv-hostengine -b : telemetry exposure surface; outage risk ; notify security team; rollback 1 min.
Drop SYS_ADMIN : DCP fields silently lost; no outage but capability degraded; confirm compute metrics not needed; rollback seconds.
Two items with "outage risk=yes" must go through change ticket . Especially "embedded to remote": switch moment old engine exits, new takes over, all field subscriptions rebuild; /metrics may return half fields. If monitoring has up alert, false positive.
6.4 Pre-Production Self-Check Script
Script does one thing: prove this exporter on your node "has all it should, where it should" . Runs on node bring-up and post-change.
#!/usr/bin/env bash set -euo pipefail IFS=$'
' ENDPOINT="${1:-http://127.0.0.1:9400/metrics}" EXPECT_GPU="${2:-8}" EXPECT_PROF="${3:-yes}" FAIL=0 say() { printf '[%s] %s
' "$1" "$2"; } fail() { say FAIL "$1"; FAIL=$((FAIL + 1)); } if ! curl -sf --max-time 5 "${ENDPOINT}" -o /tmp/dcgm_selfcheck.prom; then say FAIL "${ENDPOINT} unreachable, skipping checks"; exit 1; fi say OK "endpoint reachable" # 1. GPU count check n_gpu="$(grep -c '^DCGM_FI_DEV_GPU_UTIL' /tmp/dcgm_selfcheck.prom || true)" if [ "${n_gpu}" -ne "${EXPECT_GPU}" ]; then fail "GPU_UTIL series ${n_gpu}, expected ${EXPECT_GPU}"; else say OK "gpu series = ${n_gpu}"; fi # 2. Critical field existence for f in DCGM_FI_DEV_FB_USED DCGM_FI_DEV_GPU_TEMP DCGM_FI_DEV_XID_ERRORS DCGM_FI_DEV_POWER_USAGE; do if grep -q "^${f}" /tmp/dcgm_selfcheck.prom; then say OK "field ${f}"; else fail "missing field ${f}"; fi done # 3. DCP capability probe n_prof="$(grep -c '^DCGM_FI_PROF_' /tmp/dcgm_selfcheck.prom || true)" if [ "${EXPECT_PROF}" = "yes" ] && [ "${n_prof}" -eq 0 ]; then fail "DCP fields 0, check runAsUser=0 and CAP_SYS_ADMIN"; elif [ "${n_prof}" -gt 0 ]; then say OK "dcp fields = ${n_prof}"; else say SKIP "dcp not required"; fi # 4. Dimension self-check: grab one FB_USED value, determine MiB or bytes fb="$(grep -m1 '^DCGM_FI_DEV_FB_USED' /tmp/dcgm_selfcheck.prom | awk '{print $NF}')" if [ -z "${fb}" ]; then fail "FB_USED no value"; elif [ "${fb}" -gt 20000000000 ]; then say OK "FB_USED=${fb} unit bytes (4.9.0+)"; elif [ "${fb}" -gt 100 ]; then say OK "FB_USED=${fb} unit MiB (<=4.8.x)"; else fail "FB_USED=${fb} dimension anomaly, suspect field read failure"; fi # 5. Driver version label present if grep -q 'DCGM_FI_DRIVER_VERSION=' /tmp/dcgm_selfcheck.prom; then say OK "driver version label present"; else fail "missing driver version label, CSV may lack label line"; fi if [ "${FAIL}" -gt 0 ]; then say FAIL "selfcheck failed with ${FAIL} issue(s)"; exit 2; fi say OK "all checks passed"Sample output shows all OK. Step 4 dimension check is script's highest value: detects "your PromQL all wrong by one order of magnitude" within 3 seconds post-exporter-upgrade — otherwise discovered only when real VRAM alert fails to fire.
6.5 Operational Maxims Summary
Compress all above into seven lines — the dividing line between usable and useless monitoring:
GPU_UTIL only answers "is GPU occupied?", not "is compute saturated?". Latter requires DCGM_FI_PROF_*; any dashboard treating util as compute metric is lying.
Alert primary metric is "series count", not "value". GPU drop, init failure, permission downgrade — first symptom is a series vanishing. count by(Hostname) rule must precede all threshold rules.
Never use avg for conclusions. 8-GPU group with one straggler, mean stays >85%. Use median for deviation detection, max by / min by for localization.
Collection interval must divide scrape interval. Else rate() steps, max_over_time misses spikes, all window-based judgments become brittle.
Dimensions are version issue, not syntax issue. 4.9.0+ util becomes ratio, VRAM becomes bytes. Upgrade exporter must run dimension self-check first, else "no error but all wrong".
Enable DCP fields on-demand, per-node pools. Training nodes disable to save 1–3% throughput; inference nodes enable to distinguish compute-bound vs memory-bound.
Port 5555 bind loopback only. Host engine has no auth; exposing it = leaking GPU serials, MIG topology, process list.
GPU monitoring difficulty never lies in "install an exporter", but in twisting sampling semantics, permission model, metric dimensions, and alert grading into a self-proving evidence chain . Achieve that, and every number on Grafana becomes part of fault conclusion, not decoration.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
