Skip to content
Why nvidia-smi Lies: DCGM Exporter + Prometheus for True GPU Observability
Operations 64 min read

Why nvidia-smi Lies: DCGM Exporter + Prometheus for True GPU Observability

This article details building production-grade GPU cluster observability using DCGM Exporter and Prometheus, covering hardware counter metrics (SM_ACTIVE, DRAM_ACTIVE), Kubernetes pod mapping, MIG monitoring, alerting rules, and real-world troubleshooting cases like NVLink degradation and thermal throttling.

Ops Community
Ops Community
Ops Community

1. Why nvidia-smi Misleads: Hardware Counter Fundamentals

In large-model training and high-concurrency inference (vLLM/SGLang) clusters, engineers often see DCGM_FI_DEV_GPU_UTIL at 99% while training step time doubles or inference throughput stalls. The root cause: nvidia-smi /NVML reports the percentage of time any GPU engine (compute, memory copy, encode/decode) was non-idle in the sampling window. Even slow data movement or tiny kernels using 5% of CUDA cores can show ~100%.

True compute utilization requires NVIDIA DCGM profiling counters mapped to Kubernetes pods:

Compute-intensive (Prefill): SM_ACTIVE and TENSOR_ACTIVE — measure SM and Tensor Core saturation on GEMM.

Memory-intensive (Decode): DRAM_ACTIVE — HBM bandwidth utilization; during decode SM_ACTIVE may be 20% while DRAM_ACTIVE hits 85% physical limit.

Communication & health: NVLINK throughput and XID_ERRORS — multi-GPU tensor parallel (TP=8) depends on NVLink; CRC errors or XID faults stall the whole job.

Multi-tenant attribution: DCGM Exporter mounts /var/lib/kubelet/pod-resources/kubelet.sock to bind physical GPU UUID to pod, namespace, container labels.

1.1 Full-Chain Topology

[8x NVIDIA GPU (NVLink)]
  |
  +--> [NVML / Profiling HW counters]
  |
  v (1) Collect SM/DRAM/XID/temp/power
[DCGM 4.6.0 HostEngine]
  |
  v (2) Correlate /var/lib/kubelet/pod-resources/kubelet.sock
[DCGM Exporter 4.8.3 (:9400/metrics)]
  |  (outputs metrics with gpu, UUID, pod, namespace labels)
  v (3) HTTP Pull scrape (10s interval)
[Prometheus 3.13 LTS (TSDB)] ---> [Alertmanager]
  |
  v (4) PromQL aggregation
[Grafana GPU Observability Dashboard]

1.2 Production Baseline

Dimension | Spec | Version/Notes
--- | --- | ---
Compute Node | 8x NVIDIA H100 80GB SXM5 | 64 vCPU / 1TB RAM / dual 200Gb RoCE
OS & Driver | Ubuntu 24.04.1 LTS / Driver 570+ | Kernel 6.8.0-45 / CUDA 12.8+
Orchestration | Kubernetes v1.37.2 | Containerd 1.7.23 + NVIDIA Toolkit
Monitoring Stack | DCGM Exporter 4.8.3 + DCGM 4.6.0 | Prometheus 3.13 LTS + Grafana 11

1.3 Core Metric Definitions & Pitfalls

Metric | Hardware Meaning | Healthy Range | Common Pitfall
--- | --- | --- | ---
DCGM_FI_DEV_GPU_UTIL | Any engine non-idle time % | Rough reference only | Mistakes low-efficiency kernels for full utilization
DCGM_FI_PROF_SM_ACTIVE | SM active cycles ratio | Training/Prefill > 60% | Conflicts with external profilers (nsys)
DCGM_FI_PROF_DRAM_ACTIVE | HBM bandwidth active ratio | Decode 60%-85% | Ignoring bandwidth saturation leads to false compute deficit
DCGM_FI_DEV_XID_ERRORS | Last hardware/driver XID code | Must be 0 | Only checked in logs, not as time-series alert

1.4 MIG Monitoring Challenges

H100 80GB split into MIG instances (e.g., 1g.10gb, 3g.40gb). DCGM Exporter 4.8.3 adds GPU_I_ID (MIG instance ID) and GPU_I_PROFILE labels. Note: older drivers may not isolate DCGM_FI_PROF_DRAM_ACTIVE per MIG instance; aggregate by GPU_I_ID and UUID in Prometheus to avoid double-counting.

1.5 Hopper vs Blackwell Monitoring Requirements

Blackwell (B200/B300) introduces FP4/NVFP4 tensor engines and NVLink 5.0. Requires DCGM 4.6.0+ with DCGM Exporter 4.8.3+ and Driver 570+/580+ . Older DCGM 3.x returns Not Supported for new profiling fields.

2. Production Dual-Layer Configuration & Four-Dimensional Breakdown

2.1 Collection Layer: Custom Metrics CSV & DaemonSet

DCGM Exporter defaults lack deep profiling ( SM_ACTIVE, DRAM_ACTIVE, NVLINK throughput, throttle reasons). Mount custom /etc/dcgm-exporter/dcp-metrics-included.csv (format: DCGM_FIELD_ID, Prometheus_Metric_Type, Help). Key entries:

# 1. Basic compute & memory
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %)
DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %)
DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (MiB)
DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (MiB)
DCGM_FI_DEV_FB_TOTAL, gauge, Framebuffer memory total (MiB)

# 2. Health, temp, power, throttle, XID, ECC
DCGM_FI_DEV_GPU_TEMP, gauge, GPU temperature (C)
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (C)
DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (W)
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock (MHz)
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock (MHz)
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS, gauge, Clock throttle reasons bitmask
DCGM_FI_DEV_XID_ERRORS, gauge, Last XID error
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, counter, Total double-bit ECC errors (volatile)

# 3. Deep profiling counters (need SYS_ADMIN/CAP_SYS_ADMIN)
DCGM_FI_PROF_SM_ACTIVE, gauge, Ratio of cycles an SM has at least one warp assigned
DCGM_FI_PROF_SM_OCCUPANCY, gauge, Ratio of warps resident on an SM
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles tensor (HMMA) pipe active
DCGM_FI_PROF_DRAM_ACTIVE, gauge, Ratio of cycles device memory interface active
DCGM_FI_PROF_PCIE_TX_BYTES, counter, Bytes transmitted over PCIe
DCGM_FI_PROF_PCIE_RX_BYTES, counter, Bytes received over PCIe
DCGM_FI_PROF_NVLINK_TX_BYTES, counter, Bytes transmitted over NVLink
DCGM_FI_PROF_NVLINK_RX_BYTES, counter, Bytes received over NVLink
DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL, counter, Total NVLink CRC errors

DaemonSet uses nvcr.io/nvidia/k8s/dcgm-exporter:4.8.3-4.6.0-ubuntu22.04, mounts pod-resources socket read-only, sets DCGM_EXPORTER_KUBERNETES=true, DCGM_EXPORTER_INTERVAL=10000 (10s), runs as root with SYS_ADMIN capability, resource limits 500m CPU / 512Mi memory.

2.1.1 Prometheus Scrape Config

scrape_configs:
- job_name: "nvidia-dcgm-exporter"
  scrape_interval: 10s
  scrape_timeout: 8s
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names: [monitoring]
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
    action: keep
    regex: dcgm-exporter
  - source_labels: [__meta_kubernetes_pod_node_name]
    target_label: node
  - source_labels: [__meta_kubernetes_pod_ip]
    target_label: __address__
    replacement: "${1}:9400"

2.1.2 Alertmanager Inhibition & Routing

When a GPU triggers XID 79 (fall-off-bus) or severe thermal throttle, dozens of secondary alerts fire. Configure inhibit_rules so on-call SREs receive only root-cause P0 alerts:

inhibit_rules:
- source_matchers:
  - alertname="GpuHardwareXidErrorDetected"
  target_matchers:
  - severity=~"warning|info"
  equal: ["Hostname", "gpu"]
- source_matchers:
  - alertname="GpuThermalThrottlingActive"
  target_matchers:
  - alertname="GpuComputeInefficiencyIdleWarning"
  equal: ["Hostname", "gpu"]

2.1.3 GPU Operator Helm Values

Lock image version and custom metrics ConfigMap to avoid overwrite on Operator upgrade:

dcgmExporter:
  enabled: true
  repository: nvcr.io/nvidia/k8s
  image: dcgm-exporter
  version: "4.8.3-4.6.0-ubuntu22.04"
  config:
    name: dcgm-exporter-metrics
  env:
  - name: DCGM_EXPORTER_INTERVAL
    value: "10000"
  - name: DCGM_EXPORTER_KUBERNETES
    value: "true"
  serviceMonitor:
    enabled: true
    interval: 10s

2.2 Host OS, Driver & Fabric Layer Baseline

On 8-GPU SXM nodes, missing persistence mode or FabricManager causes DCGM to re-initialize driver context on each scrape, leading to GPU state jitter or dropouts.

# Enable NVIDIA persistence mode
systemctl enable --now nvidia-persistenced
# 8-GPU SXM NVLink nodes must run FabricManager matching driver major version
systemctl enable --now nvidia-fabricmanager

Allow non-privileged perf event sampling for DCGM profiling and increase socket buffer to prevent local TCP queue drops under high-cardinality metric bursts:

# /etc/sysctl.d/99-gpu-monitoring.conf
kernel.perf_event_paranoid = -1
net.core.somaxconn = 32768
# Apply: sysctl --system

2.2.1 Verify Device Plugin & Host Services

systemctl status nvidia-persistenced nvidia-fabricmanager --no-pager | grep -E "Loaded:|Active:"
kubectl -n kube-system get pods -l name=nvidia-device-plugin-ds -o wide | head -n 6

Expected: both services active (running), device plugin pods Running.

2.2.2 RoCEv2/InfiniBand & node_exporter

Enable InfiniBand collector in node_exporter v1.8.2+:

# /etc/systemd/system/node_exporter.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/local/bin/node_exporter --collector.infiniband --collector.netdev --collector.hwmon --web.listen-address=:9100

Verify RDMA metrics:

curl -s http://127.0.0.1:9100/metrics | grep -E "node_infiniband_port_data_(transmitted|received)_bytes_total|node_infiniband_link_downed_total" | head -n 6

Output shows link_downed_total=0 and high byte counters — healthy.

2.3 Four-Dimensional Config Breakdown

① Must-change items: nodeSelector GPU node label, tolerations matching node taints, hostPath for pod-gpu-resources (default /var/lib/kubelet/pod-resources, adjust if --root-dir customized).

② Coupled items: DCGM_EXPORTER_INTERVAL (10000 ms) must equal Prometheus scrape_interval: 10s; image tag 4.8.3-4.6.0 must match host nv-hostengine and nvidia-fabricmanager driver version exactly; DCGM_EXPORTER_KUBERNETES=true plus read-only socket mount are inseparable.

③ Prometheus storage formulas: Active series = GPU nodes × GPUs/node × enabled metrics. 100 nodes × 8 GPUs × 22 metrics = 17,600 series (×7 for MIG). Memory ~8 KB/series, disk ~1.5 bytes/sample. 10s scrape, 30-day retention ≈ 6.8 GB disk, 140 MB memory overhead.

④ Absolute red lines: Never set interval < 2000 ms (2s) — high-frequency polling contends with CUDA kernels, measured 3-5% throughput drop. Never run nsys / ncu concurrently with DCGM profiling — single hardware profiling context, causes ERR_NVGPUCTRPERM or exporter hang. Never drop DCGM_FI_DEV_XID_ERRORS and DCGM_FI_DEV_CLOCK_THROTTLE_REASONS — sole early signals for hardware faults.

2.4 Config Validation

kubectl -n monitoring rollout status daemonset/dcgm-exporter
NODE_POD=$(kubectl -n monitoring get pod -l app.kubernetes.io/name=dcgm-exporter -o jsonpath='{.items[0].metadata.name}')
kubectl -n monitoring exec "${NODE_POD}" -- curl -s http://127.0.0.1:9400/metrics | grep -E "DCGM_FI_PROF_SM_ACTIVE|DCGM_FI_DEV_FB_USED" | head -n 6

Expected output shows metrics with pod, namespace, container labels and realistic values (e.g., SM_ACTIVE=0.7842, FB_USED=74210 MiB).

3. High-Frequency One-Liners & Core PromQL

3.1 On-Node Triage Commands

1. Millisecond-level real-time SM/DRAM/NVLink for specific GPUs (trigger: suspect "100% util but zero compute"):

dcgmi dmon -i 0,1 -e 203,204,1002,1003,1005,1011,1012 -d 1000 -c 5

Output columns: GPUTL MCUTL SMACT SMOCC DRAMA NVLTX NVLRX. Example anomaly: GPUTL=99 SMACT=0.142 DRAMA=0.884 — memory-bound.

2. Check 8-GPU NVLink topology & PCIe degradation (trigger: TP=4/8 latency spike):

nvidia-smi topo -m && nvidia-smi nvlink --status | grep -E "GPU|Link|Inactive" | head -n 12

Look for SYS instead of NV18 — indicates NVLink down, fallback to CPU PCIe.

3. Throttle reasons & XID history (trigger: sudden perf drop at 80°C+):

nvidia-smi -q -d PERFORMANCE,TEMPERATURE | grep -E "GPU Current Temp|Thermal Slowdown|Power Brake|HW Slowdown"

Output may show SW Thermal Slowdown: Active — SM clock forced low.

4. Locate host process & container on a GPU :

nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_memory --format=csv,noheader

3.1.1 Hardware Diagnostics & Throttle Bitmask Decoding

5. Level 2 diagnostic (requires drained node): dcgmi discovery -l && dcgmi diag -r 2. Runs memory, PCIe bandwidth tests. Output includes PCIe Bandwidth: 52.4 GB/s baseline.

6. Decode DCGM_FI_DEV_CLOCK_THROTTLE_REASONS bitmask (e.g., value 132):

python3 -c '
val = 132
reasons = {1:"GpuIdle", 2:"AppClockSetting", 4:"SWPowerCap", 8:"HWSlowdown", 32:"SWThermal", 64:"HWThermal", 128:"HWPowerBrake"}
print([name for bit, name in reasons.items() if val & bit])
'

Result: ['SWPowerCap', 'HWPowerBrake'] — simultaneous software power cap and hardware power brake.

3.1.2 Process-Level Sampling & PCIe Gen5 Degradation

7. nvidia-smi pmon -s um -c 3 shows per-PID SM/memory/encoder/decoder usage. Example: GPU 0 SM 74%, GPU 1 SM 12% — load imbalance in tensor parallel shards.

8. Check PCIe link speed/width downgrade :

lspci -d 10de: -vvv 2>/dev/null | grep -E "3D controller|LnkSta:" | head -n 8

Healthy: Speed 32GT/s, Width x16. Degraded: Speed 8GT/s (downgraded), Width x8 (downgraded) — reseat card or check motherboard slot.

3.1.3 Driver/CUDA/FabricManager Version Consistency

nvidia-smi --query-gpu=driver_version,vbios_version --format=csv,noheader | uniq && dpkg -l | grep -E "nvidia-fabricmanager|datacenter-gpu-manager" | awk '{print $2, $3}'

All must match major.minor.patch (e.g., 570.86.15).

3.2 Production PromQL Alert Rules

Aligned with DCGM Exporter 4.8.3 metric names:

groups:
- name: ai-gpu-cluster-health
  rules:
  - alert: GpuHardwareXidErrorDetected
    expr: DCGM_FI_DEV_XID_ERRORS > 0
    for: 1m
    labels: {severity: critical}
    annotations:
      summary: "Node {{ $labels.Hostname }} GPU {{ $labels.gpu }} ({{ $labels.pod }}) XID error"
      description: "XID code {{ $value }}, isolate node immediately"
  - alert: GpuThermalThrottlingActive
    expr: DCGM_FI_DEV_GPU_TEMP > 82 or (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS > 0 and DCGM_FI_DEV_GPU_UTIL > 20)
    for: 2m
    labels: {severity: warning}
  - alert: GpuEccDoubleBitError
    expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[5m]) > 0
    for: 1m
    labels: {severity: critical}
  - alert: GpuComputeInefficiencyIdleWarning
    expr: DCGM_FI_DEV_GPU_UTIL > 80 and DCGM_FI_PROF_SM_ACTIVE < 0.25
    for: 10m
    labels: {severity: info}
  - alert: GpuMemoryNearExhaustion
    expr: (DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) * 100 > 95
    for: 5m
    labels: {severity: warning}
  - alert: GpuNvlinkCrcErrorsIncreasing
    expr: rate(DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL[5m]) > 10
    for: 3m
    labels: {severity: warning}

3.3 Dashboard PromQL Queries

# 1. Allocated physical GPUs per namespace
count by (namespace) (DCGM_FI_DEV_FB_USED{pod!=""} > 100)

# 2. Top 10 GPUs by Tensor Core utilization
topk(10, avg by (Hostname, gpu, pod) (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{pod!=""}))

# 3. Per-GPU memory allocation ratio
(DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) * 100

3.3.1 vLLM/SGLang Joint Queries

vLLM pre-allocates KV cache, so FB_USED stays >90%. Combine with vllm:gpu_cache_usage_perc:

# Idle GPUs: physical mem >90% but KV cache <20%
(DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) > 0.90
and on(pod)
vllm:gpu_cache_usage_perc < 0.20

# Decode-phase HBM bandwidth pressure
avg by (Hostname, gpu, pod) (DCGM_FI_PROF_DRAM_ACTIVE{pod!=""}) * 100

3.3.2 Recording Rules for 500+ GPU Clusters

groups:
- name: gpu-cluster-aggregations
  interval: 10s
  rules:
  - record: cluster:gpu_sm_active:avg_by_ns
    expr: avg by (namespace) (DCGM_FI_PROF_SM_ACTIVE{pod!=""})
  - record: cluster:gpu_dram_active:avg_by_ns
    expr: avg by (namespace) (DCGM_FI_PROF_DRAM_ACTIVE{pod!=""})
  - record: cluster:gpu_fb_used_ratio:by_pod
    expr: DCGM_FI_DEV_FB_USED{pod!=""} / (DCGM_FI_DEV_FB_USED{pod!=""} + DCGM_FI_DEV_FB_FREE{pod!=""})
  - record: cluster:gpu_allocated_cards:count
    expr: count(DCGM_FI_DEV_FB_USED{pod!=""} > 100)

3.3.3 Six Core Dashboard Panels

Panel | Core PromQL | Type | Focus
--- | --- | --- | ---
1. Cluster GPU Count & Ownership | count by (namespace) (DCGM_FI_DEV_FB_USED) | Pie/Stat | Team quota usage
2. SM Real Compute Activity | DCGM_FI_PROF_SM_ACTIVE | Time-series | Distinguish true saturation vs. small-kernel spin
3. HBM Bandwidth Level | DCGM_FI_PROF_DRAM_ACTIVE | Time-series | Locate decode memory bottlenecks
4. Per-GPU Memory Allocation | FB_USED / (FB_USED + FB_FREE) | Heatmap/Bar | OOM & slow leak early warning
5. Temp & Throttle Events | DEV_GPU_TEMP + THROTTLE_REASONS | Dual-axis | Detect rack cooling & power walls early
6. XID & ECC Errors | DEV_XID_ERRORS + ECC_DBE | Stat table (red >0) | Second-level GPU drop & memory fault detection

3.3.4 Skew & Zombie GPU Detection

# 1. Severe load skew within same pod (max-min SM_ACTIVE > 50%)
max by (namespace, pod) (DCGM_FI_PROF_SM_ACTIVE{pod!=""}) - min by (namespace, pod) (DCGM_FI_PROF_SM_ACTIVE{pod!=""}) > 0.50

# 2. Zombie pods: FB_USED > 10GB but 1h avg SM_ACTIVE < 2%
(DCGM_FI_DEV_FB_USED{pod!=""} > 10240) and on(pod, gpu) (avg_over_time(DCGM_FI_PROF_SM_ACTIVE{pod!=""}[1h]) < 0.02)

4. Three Deep-Water Failure Loops

4.1 Training: GPU_UTIL 100% but Step Time 2x (NVLink Degradation + Fake Saturation)

Symptom: 8x H100 node, all GPU_UTIL=99-100%, step time 1.4s → 3.1s. GpuComputeInefficiencyIdleWarning fires.

Triage: PromQL compare SM_ACTIVE vs NVLINK_TX_BYTES; run dcgmi dmon and nvidia-smi topo -m; check dmesg for XID.

dcgmi dmon -i 0-7 -e 203,1002,1011 -c 2
nvidia-smi topo -m | head -n 10
dmesg -T | grep -iE "NVRM: Xid|nvlink" | tail -n 8

Evidence: GPU 3 NVLTX=0, topo shows SYS for GPU 3 links, dmesg shows Xid 74 NVLink fatal error on link 4.

Root Cause: GPU 3 (PCIe 0000:4b:00) NVLink link failure → driver isolates link → AllReduce falls back to CPU PCIe ( SYS) → other 7 GPUs spin in CUDA Wait Kernel → GPU_UTIL=100% but SM_ACTIVE=18%.

Fix: Taint node

kubectl taint nodes gpu-node-08 hardware=nvlink-fault:NoSchedule

, save checkpoint, reboot or nvidia-smi -r -i 3, run dcgmi diag -r 3. If unrecoverable, replace GPU/board.

Quantitative Recovery:

Metric | Failure (NVLink down → SYS) | Recovery (NV18 full speed) | Improvement
--- | --- | --- | ---
DCGM_FI_PROF_SM_ACTIVE | 0.18 (18.4%) | 0.76 (76.2%) | 4.1× effective compute
Step time | 3.10 s | 1.38 s | 55.5% reduction
NVLINK_TX_BYTES (GPU 3) | 0 B/s | 215 GB/s | Full collective comms restored

4.2 DCGM Exporter Missing Pod Labels & Frequent Restarts (Socket Path Mismatch + Profiler Conflict)

Symptom: Metrics lack pod / namespace labels; exporter pods CrashLoopBackOff every few hours.

Logs:

Failed to connect to kubelet pod-resources socket: no such file or directory

and

Unable to watch field 1002: Profiling module already in use by another process

.

Root Cause: Cluster uses custom --root-dir=/data/kubelet, so socket at /data/kubelet/pod-resources but DaemonSet mounts default path. Additionally, user containers run long-lived nsys profiling, monopolizing hardware counters.

Fix: Update DaemonSet hostPath to /data/kubelet/pod-resources; ban nsys / ncu in production inference containers; schedule profiling on dedicated test nodes.

Result: Pod label completeness 0% → 100%; exporter restarts 9/24h → 0; scrape duration stable at 142 ms.

4.3 Inference: P99 TPOT 5× Spike (SW Thermal Throttle → XID 79 Fall-Off-Bus)

Symptom: vLLM TPOT 18 ms → 95 ms, GpuThermalThrottlingActive fires, then GPU 5 triggers GpuHardwareXidErrorDetected (XID 79), instance crashes.

Evidence: dmesg shows Xid 79 GPU has fallen off the bus; nvidia-smi -q reveals GPU 5 temp 89°C, power draw 310W (TDP 700W), SM clock 795 MHz (down from 1980 MHz).

Root Cause: Rack cold-aisle fan failure → intake temp rise → GPU 5 hits 89°C → SW Thermal Slowdown (clock 795 MHz) → TPOT 5× → temp approaches 92°C hardware shutdown + power transient → XID 79.

Fix: kubectl cordon node, engage IDC to repair fan, cold reboot to re-enumerate GPU 5. Lower thermal alert threshold from 85°C to 80°C, auto-drain at 83°C. Temporary power cap nvidia-smi -i 0-7 -pl 500 (500W) if cooling repair delayed.

Post-Recovery: GPU 5 temp 68°C, SM clock 1980 MHz, TPOT 18 ms. Verify ECC retired pages: nvidia-smi -i 5 -q -d PAGE_RETIREMENT — no double-bit errors, no pending retirement.

4.4 Slow Memory Leak & Fragmentation OOM (FB_USED Staircase Climb)

Symptom: Multi-modal inference pod (2x H100 80GB) runs 72h, QPS stable, then GpuMemoryNearExhaustion → CUDA out of memory restart.

PromQL 48h trend: FB_USED grows ~2.5 GB/hour (69120 → 79890 MiB) while QPS flat.

Root Cause: Image preprocessing cache lacks LRU eviction; PyTorch exception paths skip torch.cuda.empty_cache(), leaving orphaned tensor refs.

Fix: Add LRU max-size to cache; set --gpu-memory-utilization 0.88 (12% safety buffer); add PromQL slope alert deriv(DCGM_FI_DEV_FB_USED[6h]) > 0.

Result: Memory growth +2.5 GB/h → 0 MB/h (stable at 68.4 GB); uptime <72h → >30 days zero OOM.

4.5 FabricManager Missing → 8-GPU NVSwitch Init Failure

Symptom: Single-GPU PyTorch works; 8-GPU DDP/TP fails with NCCL error: NVLink initialization failed; DCGM_FI_PROF_NVLINK_TX_BYTES=0.

Triage: systemctl status nvidia-fabricmanager shows disabled, inactive (dead); nvidia-smi nvlink -s shows all 18 links Inactive.

Root Cause: SXM servers require host nvidia-fabricmanager to train NVSwitch links and push routing tables. Base image omitted systemctl enable --now nvidia-fabricmanager.

Fix: Enable and start service; verify 18 links Active; rerun all_reduce_perf benchmark.

Result: 0/18 → 18/18 links active; AllReduce bandwidth 382 GB/s (theoretical peak).

5. Automation, Risk Matrix & Engineering Iron Laws

5.1 Health Audit Script gpu_cluster_health_audit.sh

Read-only pre-join or on-alert check: persistence mode, XID errors, NVLink topology, thermal throttle, DCGM exporter port connectivity. Supports --dry-run.

5.2 Pod-Label Mapping Validation dcgm_metrics_diff_check.sh

Compares DCGM Exporter exposed pod labels against actual running GPU pods per namespace to catch "shadow containers".

5.2.1 XID Error Code Quick Reference

XID | Meaning | Attribution | Action
--- | --- | --- | ---
13/31 | Graphics Exception / MMU Fault | App CUDA OOB | Debug app/framework, no hardware swap
48 | Double Bit ECC (DBE) | Physical HBM bit failure | Drain node, <code>dcgmi diag</code>, RMA
63/64 | ECC Page Retirement Success/Fail | Bad page isolation status | 64 = spare exhausted, replace GPU
74 | NVLink Fatal Error | Physical link/bridge | Reboot retest, then replace GPU/board
79 | GPU Fallen Off Bus | Power/thermal/PCIe hard drop | Check power/cooling, cold reboot, replace if recurs
94/95 | Contained/Uncontained ECC | HBM isolation error | 95 contaminates neighbors, immediate GPU reset

5.2.2 Pre-Production Acceptance Checklist

All GPU nodes schedule dcgm-exporter with zero restarts.

Prometheus Targets: all GPU nodes UP for job nvidia-dcgm-exporter.

Simulate test XID alert (if driver supports dcgmi test --inject) to verify Alertmanager routing.

Remote write to VictoriaMetrics/Thanos with label filter (DCGM_FI_.*|vllm:.*|cluster:gpu_.*) to cut 80%+ egress cost.

5.2.3 Common DCGM Exporter 4.8.3 Startup/Collection Issues

Log Pattern | Cause | Fix
--- | --- | ---
Failed to connect to kubelet pod-resources | <code>kubelet.sock</code> path missing/mismatched | Verify <code>--root-dir</code>, correct <code>hostPath</code>
Profiling module is already in use | Conflict with <code>nsys</code>/<code>ncu</code> | Stop profiler in business container, restart exporter
nvmlInit() failed: Driver/library mismatch | Host driver upgraded, node not rebooted | Drain node, reboot to unify kernel modules
context deadline exceeded (Prometheus) | Too many GPUs/metrics per scrape | Keep 10s interval, trim non-core CSV metrics

5.2.4 Offline/air-gapped Quick Smoke Test

docker run --rm -d --name dcgm-smoke-test --gpus all --cap-add SYS_ADMIN -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:4.8.3-4.6.0-ubuntu22.04 && sleep 5 && curl -sf http://127.0.0.1:9400/metrics | grep -E "^DCGM_FI_DEV_GPU_UTIL{" && docker stop dcgm-smoke-test

5.2.5 Remote Write for Quarterly Retention

remote_write:
- url: "http://victoriametrics.monitoring.svc.cluster.local:8428/api/v1/write"
  queue_config:
    capacity: 10000
    max_shards: 32
    min_shards: 4
    max_samples_per_send: 2000
  write_relabel_configs:
  - source_labels: [__name__]
    regex: "(DCGM_FI_.*|vllm:.*|cluster:gpu_.*)"
    action: keep

5.2.6 Daily Health One-Liner

PROM_URL="http://127.0.0.1:9090/api/v1/query"
echo "=== Daily GPU Cluster Health ==="
echo -n "1. Allocated GPUs: " && curl -sf "${PROM_URL}?query=count(DCGM_FI_DEV_FB_USED{pod!=\"\"}>100)" | jq -r '.data.result[0].value[1] // "0"'
echo -n "2. Avg SM_ACTIVE: " && curl -sf "${PROM_URL}?query=avg(DCGM_FI_PROF_SM_ACTIVE{pod!=\"\"})" | jq -r '.data.result[0].value[1] // "0"'
echo -n "3. Hot GPUs (>82C): " && curl -sf "${PROM_URL}?query=count(DCGM_FI_DEV_GPU_TEMP>82)" | jq -r '.data.result[0].value[1] // "0"'
echo -n "4. XID Error GPUs: " && curl -sf "${PROM_URL}?query=count(DCGM_FI_DEV_XID_ERRORS>0)" | jq -r '.data.result[0].value[1] // "0"'
# Output example:
# 1. Allocated GPUs: 612
# 2. Avg SM_ACTIVE: 0.6841
# 3. Hot GPUs (>82C): 0
# 4. XID Error GPUs: 0

Weekly CronJob runs dcgmi diag -r 2 on idle nodes to catch latent memory faults and PCIe aging.

5.3 Production Change Risk Matrix

Change | Impact & Window | Pre-checks & Backup | Canary & Rollback
--- | --- | --- | ---
Update DCGM metrics CSV | Exporter restart; no business impact | Backup ConfigMap YAML | Single node test; revert ConfigMap
Scrape interval < 5s | Slight GPU register load increase | Ensure no <code>nsys</code> running | Canary 1 node; revert to 10000 ms
<code>nvidia-smi -r</code> GPU reset | All GPUs on node interrupted | <code>kubectl drain</code> node first | Single GPU validate; else host reboot
<code>nvidia-smi -pl</code> power cap | Reduces peak TFLOPS | Record default power limit | Per-GPU step down; <code>-pl</code> to restore

5.3.1 Monthly Chargeback & Utilization PromQL

# 30-day avg GPUs per namespace
avg_over_time(sum by (namespace) (count by (namespace, Hostname, gpu) (DCGM_FI_DEV_FB_USED{pod!=""} > 100))[30d:1h])
# 30-day avg SM utilization per namespace (identify chronic low-efficiency teams)
avg_over_time(avg by (namespace) (DCGM_FI_PROF_SM_ACTIVE{pod!=""})[30d:1h]) * 100

Export to CSV for finance:

curl -sf "http://127.0.0.1:9090/api/v1/query?query=..." | jq -r '.data.result[] | [.metric.namespace, .value[1]] | @csv'
# Output:
"ai-inference","128.42"
"llm-pretrain","448.10"
"algo-sandbox","35.48"

5.4 Four Engineering Iron Laws

Always judge true utilization with SM_ACTIVE + DRAM_ACTIVE : GPU_UTIL only tells if the card is "doing something"; tensor core and memory bandwidth reveal whether the model is compute-bound or memory-bound.

XID errors & ECC double-bit must be P0 real-time alerts : Any XID 31/48/74/79 or ECC_DBE increase is a hardware crash precursor; auto-isolate node before mass job failures.

Lock scrape interval at 10s and align with Prometheus : Avoids micro-perturbation from millisecond polling on training throughput while keeping rate() smooth.

Must wire pod-resources socket label mapping : Without namespace and pod dimensions, physical GPU monitoring is unactionable for chargeback and fault attribution in multi-tenant clusters.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

ObservabilityKubernetesPrometheusTroubleshootingAI infrastructureGPU monitoringNVLinkDCGM Exporter
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.