Hands‑On nvidia‑smi Guide: Diagnosing GPU Utilization and Memory Usage Anomalies

1. Record driver, GPU, and tool versions

Note driver version, CUDA Driver API version, GPU model, UUID, and firmware status. The CUDA Version field shows the highest CUDA version the driver supports, not the version installed in the Python environment.

#!/usr/bin/env bash
set -euo pipefail
nvidia-smi
nvidia-smi --query-gpu=index,uuid,name,driver_version,pci.bus_id,memory.total --format=csv,noheader
uname -a

GPU indices may change after reboot, device filtering, or MIG reconfiguration; use UUID or PCI Bus ID for long‑term correlation.

2. Understand utilization vs. memory metrics

GPU utilization

reflects the proportion of time the GPU engines are busy within a sampling window, while memory utilization indicates activity of the memory engines, not the percentage of memory occupied. FB memory used shows allocated device memory. After model loading, memory can be high while GPU utilization stays near zero, which is normal while waiting for requests.

nvidia-smi --query-gpu=index,uuid,name,utilization.gpu,utilization.memory,memory.used,memory.free,memory.total,temperature.gpu,power.draw,pstate --format=csv

A single snapshot can miss short spikes; nvidia-smi dmon samples per second and can monitor power, utilization, clocks, memory, ECC, and throughput. Continuous 100% utilization is a fault clue only when accompanied by throughput drop, latency increase, temperature/power limits, Xid errors, or queue growth.

nvidia-smi dmon -s pucvmet -d 1 -o DT

3. Identify the process holding memory

Query compute processes via NVML. used_memory may appear as N/A for some MIG or driver modes and should not be forced to zero.

nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_memory --format=csv

Map the PID to user, parent process, start time, and full command line (which may contain tokens or model paths). Handle sensitive information appropriately.

PID="<PROCESS_PID>"
ps -o user,pid,ppid,lstart,etime,stat,%cpu,%mem,args -p "$PID"
pstree -aps "$PID"
cat "/proc/$PID/cgroup"

nvidia-smi pmon

shows SM, memory engine, encoder/decoder usage per process, helping decide whether a process truly consumes memory.

nvidia-smi pmon -i <GPU_ID> -s um -d 1

Short kernels may finish between samples, so missing entries in pmon do not guarantee no execution; correlate with application logs, scheduler traces, and dmon timelines.

4. Map processes in containers and Kubernetes

The host nvidia-smi shows host PIDs. Inside Docker, use cgroup files and container process listings to trace ownership.

PID="<PROCESS_PID>"
cat "/proc/$PID/cgroup"
docker ps --no-trunc --format '{{.ID}}\t{{.Names}}\t{{.Status}}'
docker top <CONTAINER_ID> -eo pid,ppid,user,etime,args

Do not kill a process merely because its name is python; a node may run training, inference, monitoring, and platform daemons simultaneously.

In Kubernetes, first locate the GPU pod on the node, then verify the container ID. All kubectl commands should explicitly specify the namespace.

kubectl get pod -n <NAMESPACE> -o wide --field-selector spec.nodeName=<NODE_NAME>
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.containerID}{"
"}{end}'

If the business namespace is unknown, discover it via platform asset inventories or scheduler records; avoid unbounded cluster exports. Device plugins and the NVIDIA GPU Operator usually reside in a platform namespace and must be inspected separately.

kubectl get pod <POD_NAME> -n <NAMESPACE> -o json | jq '{node:.spec.nodeName,containers:[.spec.containers[]|{name,resources}],status:.status.phase}'
kubectl get events -n <NAMESPACE> --field-selector involvedObject.name=<POD_NAME> --sort-by=.lastTimestamp

5. Empty process list but abnormal memory or utilization

Check whether any device file is still opened by a process. fuser and lsof need sufficient privileges and may be slow on hosts with many processes.

sudo fuser -v /dev/nvidia* 2>/dev/null || true
sudo lsof /dev/nvidia0 /dev/nvidiactl /dev/nvidia-uvm 2>/dev/null || true

If nvidia-smi shows 100% utilization but 0 MiB memory and an empty process table, continuously sample dmon / pmon, and examine Xid, driver logs, container exit residues, and virtualization layers. A single snapshot may fall within a context‑destruction or telemetry‑delay window.

for i in $(seq 1 20); do
  date -Is
  nvidia-smi --query-gpu=index,utilization.gpu,memory.used,pstate,power.draw --format=csv,noheader
  sleep 1
done

If repeated sampling still shows anomalies, inspect kernel and driver logs. Xid codes indicate different failure modes; seeing an Xid does not automatically mean hardware damage.

sudo journalctl -k --since '-30 min' --no-pager | grep -Ei 'NVRM|Xid|nvidia|nvlink|pcie' || true
dmesg -T | grep -Ei 'NVRM|Xid' | tail -n 100 || true

6. Check temperature, power, clock, and throttling

When performance degrades, verify whether the GPU is in a low P‑State, has hit power or thermal limits, and whether SM or memory clocks match the load. Query fields vary with driver version; list available fields first.

nvidia-smi --help-query-gpu | less
nvidia-smi -q -d TEMPERATURE,POWER,CLOCK,PERFORMANCE -i <GPU_ID>

High temperature alone does not guarantee throttling; power near the limit is not necessarily abnormal. Only when clocks drop, limit reasons activate, and throughput falls together with temperature/power changes can a solid evidence chain be built.

nvidia-smi --query-gpu=timestamp,index,pstate,temperature.gpu,power.draw,power.limit,clocks.sm,clocks.mem,utilization.gpu --format=csv -l 1

7. Examine ECC, PCIe, and NVLink

ECC, PCIe replay, and NVLink errors can cause performance or stability issues; supported fields depend on GPU model and driver.

nvidia-smi -q -d ECC,PAGE_RETIREMENT -i <GPU_ID>
nvidia-smi --query-gpu=index,ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv

Non‑zero cumulative ECC does not equal a current fault; observe volatile vs. aggregate counts, growth over time, and Xid. Uncorrectable errors require NVIDIA and hardware‑vendor procedures.

nvidia-smi topo -m
nvidia-smi topo -p2p r
numactl --hardware

NVLink status commands work only on NVLink‑capable devices; an error on unsupported hardware should not be interpreted as a link failure.

nvidia-smi nvlink --status -i <GPU_ID>
nvidia-smi nvlink --capabilities -i <GPU_ID>

8. Check MIG and device partitioning

When MIG is enabled, a physical GPU is split into multiple GPU Instances and Compute Instances. Memory, utilization, and process information must be interpreted per MIG device; container‑visible UUIDs may be MIG UUIDs.

nvidia-smi -q -d MIG
nvidia-smi mig -lgi
nvidia-smi mig -lci

Creating, destroying, or disabling MIG affects all instances on the physical GPU and usually requires draining workloads first. For routine troubleshooting, view‑only queries are sufficient; do not modify MIG layout merely to “refresh” state.

9. Verify driver modules and device health

Confirm NVIDIA kernel module version, device nodes, and persistence daemon status. Mismatched host driver and container CUDA user‑space can cause CUDA initialization or symbol errors in application logs.

modinfo nvidia | grep -E '^(version|filename):'
lsmod | grep '^nvidia'
ls -l /dev/nvidia*
systemctl status nvidia-persistenced --no-pager 2>/dev/null || true

Inside a container, verify driver visibility using an approved CUDA base image.

docker run --rm --gpus all <TRUSTED_CUDA_IMAGE> nvidia-smi --query-gpu=index,uuid,name,driver_version --format=csv

If the host succeeds but the container fails, first check the NVIDIA Container Toolkit, Docker device request, CDI configuration, and container permissions before reinstalling drivers.

10. Diagnose OOM and memory fragmentation

CUDA OOM does not always mean nvidia-smi shows 100% memory usage. Framework caches, reserved memory, fragmentation, contexts, communication buffers, and transient peaks can cause allocation failures. Preserve the full application error stack before collecting process and GPU state.

PID="<PROCESS_PID>"
nvidia-smi --query-gpu=index,memory.used,memory.free,memory.total --format=csv
grep -E 'VmRSS|VmSwap|Threads' "/proc/$PID/status"
journalctl -k --since '-15 min' --no-pager | grep -Ei 'oom|killed process' || true

System OOM Killer and CUDA OOM are different layers; the former kills host processes and logs in the kernel, while the latter returns device‑memory‑allocation errors from the framework. Conclusions must cite the corresponding logs.

11. Safely terminate faulty tasks

After confirming PID, user, container, job ID, checkpoint, and business impact, prefer graceful termination via the scheduler or application. SIGTERM allows the program to clean up; only after a timeout consider SIGKILL.

PID="<PROCESS_PID>"
ps -fp "$PID"
sudo kill -TERM "$PID"
for i in $(seq 1 30); do
  kill -0 "$PID" 2>/dev/null || exit 0
  sleep 1
done
echo "process did not exit after SIGTERM" >&2
exit 1

SIGKILL

instantly kills the process, potentially corrupting output, losing checkpoints, or hanging other ranks in distributed jobs. Use it only when graceful exit is impossible and a recovery plan exists.

sudo kill -KILL <PROCESS_PID>
nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_memory --format=csv

12. Persistence mode and GPU reset boundaries

Persistence mode reduces driver initialization overhead when no client is attached but does not fix a stuck kernel. Changing it alters device runtime policy; record the original value and be able to roll back.

nvidia-smi --query-gpu=index,persistence_mode --format=csv
sudo nvidia-smi -pm 1 -i <GPU_ID>
nvidia-smi --query-gpu=index,persistence_mode --format=csv

If the change does not behave as expected, restore the original state.

sudo nvidia-smi -pm 0 -i <GPU_ID>
nvidia-smi --query-gpu=index,persistence_mode --format=csv

GPU reset terminates or corrupts all contexts on the GPU; some NVLink topologies, MIG configurations, display devices, or virtualized environments do not support single‑GPU reset. Before resetting, drain tasks, confirm no processes hold the device, back up data, and prepare a node‑reboot plan.

sudo fuser -v /dev/nvidia<GPU_ID> 2>/dev/null || true
sudo nvidia-smi --gpu-reset -i <GPU_ID>
nvidia-smi -i <GPU_ID>

If reset fails, do not loop; do not unload the driver module while it is in use. Preserve Xid, topology, process, and driver logs, then isolate the node or schedule a maintenance reboot.

13. Automated fault snapshot collection

The following script creates a timestamped directory and saves a comprehensive GPU overview, queried fields, topology, process list, kernel logs, and system information. It is read‑only and does not terminate processes or reset GPUs.

#!/usr/bin/env bash
set -euo pipefail
OUT_DIR="<DIAGNOSTIC_DIR>/gpu-$(date +%Y%m%d-%H%M%S)"
install -d -m 0700 "$OUT_DIR"

nvidia-smi > "$OUT_DIR/nvidia-smi.txt"
nvidia-smi -q > "$OUT_DIR/nvidia-smi-q.txt"
nvidia-smi topo -m > "$OUT_DIR/topology.txt"
nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_memory --format=csv > "$OUT_DIR/processes.csv"
journalctl -k --since '-60 min' --no-pager > "$OUT_DIR/kernel.log"
uname -a > "$OUT_DIR/uname.txt"

Limit permissions on the diagnostic directory because it may contain usernames, commands, containers, and internal topology. Follow your organization’s data‑handling policy when handing off the collected data.

14. Monitoring instead of manual screenshots

Production GPU monitoring typically uses DCGM Exporter or an NVML collector. Prometheus metric names follow the exporter’s exposed names. Observe GPU utilization, memory, temperature, power, clocks, Xid, ECC, PCIe/NVLink, and business throughput together.

#!/usr/bin/env bash
set -euo pipefail
EXPECTED_GPUS="<EXPECTED_GPU_COUNT>"
ACTUAL_GPUS="$(nvidia-smi --query-gpu=index --format=csv,noheader | wc -l)"

nvidia-smi --query-gpu=timestamp,index,uuid,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw,pstate --format=csv

if [[ "$ACTUAL_GPUS" -ne "$EXPECTED_GPUS" ]]; then
  echo "GPU count mismatch: expected=$EXPECTED_GPUS actual=$ACTUAL_GPUS" >&2
  exit 2
fi

Final judgment should align GPU metrics with the application phase: model loading (memory growth), inference (high memory, low utilization until requests arrive), communication (NVLink/PCIe activity), or idle/waiting (low utilization). Only when metrics diverge from the expected phase and are supported by process, log, Xid, temperature, or performance evidence should an anomaly be declared.

15. Normal patterns for different workloads

Training jobs usually show stable high memory, high SM utilization, periodic communication, and I/O spikes during checkpointing. Inference services may keep most memory allocated after model load but only raise utilization when handling requests. Video encode/decode workloads primarily use encoder/decoder engines, so GPU‑Util may not fully reflect load. Multi‑process services can switch quickly between processes. Before troubleshooting, identify whether the application is in loading, warm‑up, compute, communication, data‑wait, save, or idle stage.

If one card’s utilization is markedly lower than its peers, check whether it sits at a pipeline boundary, is waiting for the slowest rank, has uneven batch sizes, or is blocked on CPU data loading. A single card at 0 % does not necessarily indicate hardware failure; if its memory, process list, and NVLink status match the others, application logs may explain the wait. Conversely, sustained 100 % utilization with zero business throughput warrants checking for kernel hangs, communication retries, error‑recovery loops, or driver Xid events.

16. Detect continuous memory growth

Memory growth can be normal (cache warm‑up, KV‑Cache expansion, CUDA Graph capture) or indicate unreleased requests, tensor reference leaks, or stray tasks. Run a fixed request set repeatedly, recording used memory, framework cache stats, and active request count after each round. If memory does not drop after the workload becomes idle, it may still be a reusable cache (e.g., PyTorch) rather than a leak. True leaks manifest as unrecoverable allocation failures, growing used memory, and inability to free memory across rounds.

When memory growth eventually triggers OOM, retain the first error stack. Subsequent OOMs may be cascading effects (e.g., one rank fails, others error on communication). Do not only capture the final "CUDA out of memory" line; record the rank that first failed, allocation size, free memory at that moment, input length, and concurrency.

17. Driver upgrades and node isolation

Driver upgrades affect all GPU tasks, kernel modules, and container compatibility and should not be the first response to a routine fault. Upgrade only when error codes, compatibility matrices, known defects, or vendor recommendations point to a driver issue. Before upgrading, drain the node, checkpoint jobs, record current driver and firmware versions, verify the new driver’s CUDA and container image compatibility, and prepare a rollback package and maintenance reboot plan.

If a GPU repeatedly shows uncorrectable ECC, Xid, PCIe/NVLink errors, or fails to recover after reset, isolate the node and prevent new scheduling rather than allowing the scheduler to keep retrying. Isolation conclusions must cite GPU UUID, timestamps, error codes, kernel logs, and affected jobs; after recovery, perform stress verification before returning the node to service.

18. Alert thresholds must align with business baselines

High GPU utilization alone is not a good alert; training and high‑throughput inference are expected to keep the device busy. Effective alerts combine business‑throughput drops or queue growth with deviations in GPU utilization, clocks, temperature, power, or error states. Similarly, memory usage should not be fixed‑percentage based; inference services often pre‑allocate KV‑Cache, so sustained high memory can be normal, while sudden drops may indicate worker termination or model unload.

Temperature, power, and ECC thresholds should be set according to the GPU datasheet, datacenter environment, and hardware team policies. Verify that Prometheus metric names match the actual exporter, and that GPU UUID, node, pod, and container labels are correctly attached to avoid aggregating multiple cards into a single misleading metric.

When defining recovery windows, remember that process exit, memory release, driver telemetry, and scheduler state synchronization may take time. An alert that clears on a single sample can mask jitter. Correlate GPU metrics, application requests, job status, node events, and Xid on a unified timeline, and document whether recovery involved task stop, GPU reset, node reboot, or simply waiting for load to finish.