Kubernetes GPU Operations: Drivers, Device Plugins & Resource Scheduling Deep Dive

Introduction: A Real-World Pending Incident

The article opens with a 2 AM alert: 12 PyTorch training jobs stuck in Pending with 0/24 nodes are available: 24 Insufficient nvidia.com/gpu. Physical GPUs were idle (only 2 of 8 A100s used, <30% memory), but Kubernetes allocatable showed 0. Root cause: kubelet restarted three days prior, but the NVIDIA device plugin DaemonSet pod did not restart; its gRPC socket remained, yet the ListAndWatch stream was broken. Kubelet retained the last device snapshot, marking all 8 GPUs as allocated.

This exposes the three-layer dependency chain:

Kernel driver (nvidia.ko) → Container runtime (CDI / nvidia-container-runtime) → Device plugin (kubelet resource ledger)

Any layer misalignment yields "GPU unusable" but with distinct root causes and fixes.

Applicability & Environment

Applicable Scenarios

Self-managed Kubernetes (kubeadm, RKE2, k3s, k0s, MicroK8s) running AI training/inference.

Need GPU partitioning: exclusive, MIG hard partitioning, time-slicing soft sharing.

Unified monitoring/alerting with Prometheus + DCGM Exporter.

Driver version governance: upgrade, rollback, canary, batched node maintenance.

Migration from traditional Device Plugin to DRA (Dynamic Resource Allocation).

Non-Applicable

Serverless/managed inference services (cloud provider abstracts resources).

vGPU (hypervisor-level, license server).

Windows nodes (different driver model, WDDM).

Non-NVIDIA accelerators (Ascend, Cambricon, Hygon DCU) – different resource names.

Validated Versions

Component          Version              Notes
Kubernetes         v1.34.x / v1.35.x    DRA structured params GA in 1.34; GPU Operator 26.x min 1.29
Container Runtime  containerd 2.0+      GPU Operator 26.3.0+ requires containerd 2.0
GPU Operator       v26.7.2              Default driver 595.91.07, optional 615.71.09
NVIDIA Driver      595.91.07 / 615.71.09 580+ required for DRA workflows
Device Plugin      v0.20.1              Bundled with Operator 26.7.x
DCGM / Exporter    4.6.1 / 4.6.1-4.8.4  Metric names per this version
GPU Models         A100-SXM4-80GB, H100-SXM-80GB, L40S  Consumer GeForce lacks MIG
OS                 Ubuntu 22.04 / Rocky Linux 9  Kernel 5.15+, UEFI Secure Boot needs MOK signing

Three critical version checks: driver↔CUDA mapping (use nvidia-smi CUDA Version), CUDA 12/13 compatibility (CUDA 13 drops Volta and earlier), DRA vs ClusterPolicy mutual exclusion (GPUCluster CRD cannot coexist with ClusterPolicy).

Core Knowledge: How GPU Becomes a Schedulable Resource

Layer 1: Kernel Driver

Driver loads nvidia.ko, nvidia_uvm.ko, nvidia_drm.ko, nvidia_modeset.ko; creates /dev/nvidia* nodes; exposes management via nvidia-smi and NVML. Verification steps:

# 1. Modules loaded
lsmod | grep -E '^nvidia'
# 2. Device nodes exist with correct major/minor
ls -l /dev/nvidia*
# 3. NVML enumerates devices (hard evidence)
nvidia-smi -L
# 4. Driver version matches kernel module
cat /proc/driver/nvidia/version
modinfo nvidia | grep -E '^(version|filename)'

Common anomalies guide diagnosis (e.g., command not found, failed to communicate, No devices, Unknown Error). Open kernel modules ( nvidia-open) supported for Turing+ datacenter GPUs via driver.useOpenKernelModules=true; cannot mix with proprietary. Secure Boot requires mokutil --import of public key before driver install.

Layer 2: Container Runtime

Two mechanisms to expose GPUs to containers:

Mechanism 1: nvidia-container-runtime (legacy)

Wraps runc, uses libnvidia-container and NVIDIA_VISIBLE_DEVICES. Containerd config snippet (1.x path shown):

[plugins.'io.containerd.grpc.v1.cri'.containerd]
default_runtime_name = "nvidia"
[plugins.'io.containerd.grpc.v1.cri'.containerd.runtimes.nvidia]
runtime_type = "io.containerd.runc.v2"
runtime_path = "/usr/bin/nvidia-container-runtime"
privileged = false
[plugins.'io.containerd.grpc.v1.cri'.containerd.runtimes.nvidia.options]
BinaryName = "/usr/bin/nvidia-container-runtime"

Prefer nvidia-ctk runtime configure to generate correct config; restart containerd (high-risk, requires maintenance window).

Mechanism 2: CDI (Container Device Interface)

Modern standardized approach: runtime reads /var/run/cdi/nvidia.yaml (or /etc/cdi/*.yaml), containers request devices via CDI IDs. Generate spec:

sudo nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml
nvidia-ctk cdi list

Output includes nvidia.com/gpu=0, nvidia.com/gpu=GPU-3f7c1a2e-..., enabling UUID and MIG instance references. GPU Operator 26.x defaults to CDI, requiring containerd 2.0+.

Determine active path:

ls -l /var/run/cdi/ 2>/dev/null
containerd config dump 2>/dev/null | grep -A3 'default_runtime_name'
ctr -n k8s.io run --rm --gpus 0 docker.io/nvidia/cuda:12.4.1-base-ubuntu22.04 t1 nvidia-smi -L

ctr run --gpus

fails on legacy runtime with no CDI devices found.

Layer 3: Device Plugin & Kubelet Resource Ledger

Kubernetes knows only extended resources ( <vendor-domain>/<resource-name>). Device plugin (DaemonSet) registers via /var/lib/kubelet/device-plugins/nvidia-gpu.sock, performs Register, establishes ListAndWatch stream, and handles Allocate requests.

Registration Flow

Create Unix socket in plugin directory.

Call Register via kubelet.sock with socket name, API version (v1beta1), resource name ( nvidia.com/gpu).

Establish ListAndWatch long-lived stream pushing device list and health.

On Allocate, return device nodes, mounts, env vars, CDI device IDs.

Kubelet writes reported device count into Node status.capacity and status.allocatable. Capacity comes from plugin report, not real-time nvidia-smi – hence the opening incident.

Four Hard Rules of Extended Resources

Integer only ( 0.5 rejected).

Non-compressible (no kubelet eviction pressure).

No overcommit (scheduler strictly counts allocatable).

Request must equal limit (auto-equalized).

Consequences: No VRAM isolation (time-slicing shares VRAM, one OOM kills all); GPU count doesn't affect QoS (QoS still from CPU/memory).

Allocate Response

Returns env vars ( NVIDIA_VISIBLE_DEVICES, NVIDIA_DRIVER_CAPABILITIES=compute,utility), mounts ( /usr/bin/nvidia-smi, etc.), device nodes, CDI device IDs. Driver libraries ( libnvidia-ml.so.*, libcuda.so.*) injected by runtime/CDI, not device plugin – crucial for troubleshooting.

Health Checks & Device Status

ListAndWatch includes Health (Healthy/Unhealthy). Plugin polls NVML; unhealthy devices removed from allocatable. Capacity stays at 8, allocatable drops to 7 – always check allocatable.

kubectl get node gpu-node-01 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'

GFD: GPU Feature Discovery

Labels nodes with GPU attributes (product, memory, compute capability, MIG capability). Enables scheduling like:

nodeSelector:
  nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3

Label values replace spaces with hyphens; copy from node labels to avoid typos. nvidia.com/gpu.memory (MiB per GPU) enables memory-based affinity:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nvidia.com/gpu.memory
          operator: Gt
          values: ["40000"]

MIG: Hardware-Level Partitioning

MIG (Ampere+) splits GPU into independent instances with dedicated SMs, memory, L2 slices – fault-domain isolation . A100-80GB profiles:

Profile    SMs    Memory    Typical Use
1g.10gb    1/7    10 GB     Small inference, notebooks
2g.20gb    2/7    20 GB     Medium inference
3g.40gb    3/7    40 GB     Fine-tuning
4g.40gb    4/7    40 GB     Memory-constrained training
7g.80gb    7/7    80 GB     Full GPU equivalent

Query supported profiles: nvidia-smi mig -lgip. Two plugin strategies:

single : all MIG instances as nvidia.com/gpu (simple, no spec differentiation).

mixed : per-profile resources ( nvidia.com/mig-1g.10gb: 7, nvidia.com/mig-3g.40gb: 1) – recommended for production.

MIG constraints: reconfiguration requires draining all processes on that GPU; config lost on driver reload/reboot (needs MIG Manager); MIG instances don't support time-slicing; DCGM metric semantics change per instance.

Time-Slicing & MPS: Soft Sharing

Time-slicing virtualizes one GPU into N replicas (e.g., replicas: 4 → 8 GPUs report 32 nvidia.com/gpu). Config via ConfigMap:

version: v1
flags:
  migStrategy: none
sharing:
  timeSlicing:
    renameByDefault: false
    failRequestsGreaterThanOne: true
    resources:
    - name: nvidia.com/gpu
      replicas: 4

Key params: replicas (shares per GPU), renameByDefault (use nvidia.com/gpu.shared to distinguish from exclusive nodes), failRequestsGreaterThanOne (reject pods requesting >1 share). Apply via node label nvidia.com/device-plugin.config=any.

Risks: No VRAM isolation (one OOM kills all), no fault isolation (Xid reset kills all), no compute guarantee (latency interference), requesting 2 shares ≠ 50% compute .

MPS (Multi-Process Service) provides user-space compute/memory quotas, better isolation but experimental in device plugin, mutually exclusive with MIG.

DRA: Next-Gen Scheduling Model

Device Plugin limitation: resource is just a count. DRA introduces structured requests: DeviceClass: device category (e.g., gpu.nvidia.com). ResourceSlice: node's device inventory with attributes. ResourceClaim: specific request with CEL expression selectors. ResourceClaimTemplate: template for pods.

Example claim for H100:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: h100-only
  namespace: ai-infer
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
          count: 1
          selectors:
          - cel:
              expression: |
                device.attributes["gpu.nvidia.com"].productName == "NVIDIA-H100-80GB-HBM3"

Pod references claim via resources.claims. DRA operational differences: no nvidia.com/gpu scalar; query kubectl get resourceslice; cannot run alongside Device Plugin; requires containerd CDI + driver 580+; GPUCluster CRD mutually exclusive with ClusterPolicy – migration is rebuild, not toggle. Recommendation: 2026 still Device Plugin primary, DRA for new clusters/workloads.

Troubleshooting Methodology: Layer-by-Layer

Layer 0  Hardware & PCIe          → lspci / dmesg / nvidia-smi -L
Layer 1  Kernel driver            → lsmod / nvidia-smi / NVML
Layer 2  Container runtime        → ctr run --gpus / CDI spec
Layer 3  Device plugin            → DaemonSet logs / kubelet logs / socket
Layer 4  Resource reporting       → Node allocatable / GFD labels
Layer 5  Scheduling & admission   → Pod events / scheduler logs / ResourceQuota
Layer 6  In-container runtime     → nvidia-smi / CUDA errors

Single-point test:

sudo ctr -n k8s.io run --rm --gpus 0 docker.io/nvidia/cuda:12.4.1-base-ubuntu22.04 gpuchk nvidia-smi -L

. Success → layers 0-2 OK, issue in layer 3+. Failure → fix driver/runtime first.

Deployment Option A: GPU Operator (Recommended)

Prerequisites & Pre-Check Script

Cluster ≥1.29, GPU bare metal or passthrough VMs, access to nvcr.io / helm.ngc.nvidia.com (or internal mirror), maintenance window for reboots, Prometheus Operator deployed. Pre-check script validates PCI visibility, no nouveau, Secure Boot status, kernel headers, containerd version, kubelet plugin dir, disk space, swap/hugepages.

Installation (Helm)

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --version v26.7.2 \
  --set driver.enabled=true \
  --set driver.version=595.91.07 \
  --set toolkit.enabled=true \
  --set devicePlugin.enabled=true \
  --set dcgmExporter.enabled=true \
  --set migManager.enabled=true \
  --set gfd.enabled=true \
  --set operator.defaultRuntime=containerd \
  --set daemonsets.tolerations[0].key=nvidia.com/gpu \
  --set daemonsets.tolerations[0].operator=Exists \
  --set daemonsets.tolerations[0].effect=NoSchedule \
  --wait --timeout 30m

Parameter table explains each flag. Tolerations needed for GPU node taints ( nvidia.com/gpu=present:NoSchedule).

k3s/RKE2/k0s Differences

Containerd config paths differ; must set CONTAINERD_CONFIG, CONTAINERD_SOCKET, CONTAINERD_RUNTIME_CLASS, CONTAINERD_SET_AS_DEFAULT env vars in toolkit container. Wrong path creates empty config, restarting containerd kills all node containers.

Rollout Phases & Verification

node-feature-discovery → CPU/kernel labels

gpu-feature-discovery → GPU labels (needs driver ready)

nvidia-driver-daemonset → compile/install driver, reboot required

nvidia-container-toolkit → rewrite containerd config, restart

nvidia-device-plugin → register resources

dcgm-exporter → expose metrics

mig-manager → apply MIG config

Monitor with kubectl get pod -n gpu-operator -o wide, kubectl get ds -n gpu-operator, driver logs. Success shows nvidia-smi table in driver logs. Node labels indicate state ( nvidia.com/gpu.deploy.driver=true, etc.). Disable per-node via label nvidia.com/gpu.deploy.driver=false.

Deployment Option B: Manual (No Operator)

1. Install Driver

Ubuntu 22.04 + runfile 595.91.07: install build deps, disable nouveau, reboot, run

sh NVIDIA-Linux-x86_64-595.91.07.run --silent --dkms --no-drm --disable-nouveau --no-questions --ui=none

. Verify full nvidia-smi table.

2. Container Toolkit & Runtime Config

Add NVIDIA repo, install nvidia-container-toolkit, generate CDI spec ( nvidia-ctk cdi generate), configure containerd (

nvidia-ctk runtime configure --runtime=containerd --set-as-default

), restart containerd (cordon first).

3. Device Plugin Manifest

DaemonSet with hostPath to /var/lib/kubelet/device-plugins and /var/run/cdi, env MIG_STRATEGY=mixed, FAIL_ON_INIT_ERROR=true, securityContext drop ALL, resource limits. Ensure hostPath matches kubelet --root-dir.

Verification Matrix

1. Node Resources

kubectl get nodes -o custom-columns=NODE:.metadata.name,GPU_CAP:.status.capacity.nvidia\.com/gpu,GPU_ALLOC:.status.allocatable.nvidia\.com/gpu

Expect CAP=ALLOC=8. CAP present but ALLOC=0 → device health or plugin stream issue.

2. Node Labels

kubectl get node gpu-node-01 -o json | jq -r '.metadata.labels | to_entries[] | select(.key|startswith("nvidia.com")) | "\(.key)=\(.value)"'

Must have nvidia.com/gpu.product and nvidia.com/gpu.count.

3. Minimal Smoke Pod

Pod with nvidia.com/gpu: 1, runs nvidia-smi -L and query. Wait for Succeeded phase, check logs for GPU enumeration.

4. Real Framework Test

Run PyTorch matmul via kubectl run with GPU limit; verify device name and computation output.

5. MIG / Time-Slicing Resources

Check allocatable for nvidia.com/mig-* or multiplied nvidia.com/gpu count.

Scheduling Practices

Exclusive GPU for Training

Job with nvidia.com/gpu: 8, matching nproc_per_node=8, large /dev/shm (emptyDir Memory), CPU/memory scaled per GPU (8-16 cores, 64-128GB per GPU).

Memory-Based Selection

Use nodeAffinity with nvidia.com/gpu.memory (MiB, per GPU). Note: Gt is strict; use Gt 23000 for "≥24GB". Prefer business labels ( gpu-tier=large) over raw numeric comparisons.

Topology Awareness

Check nvidia-smi topo -m for NVLink (NV18), SYS (cross-NUMA), PHB/PXB (PCIe). Multi-GPU jobs need same interconnect domain; use topology-aware scheduler or label nodes by domain.

Resource Quotas & Multi-Tenancy

ResourceQuota

must specify both requests.nvidia.com/gpu and limits.nvidia.com/gpu. LimitRange caps per-container GPU (e.g., max 8, default 1).

Priority & Preemption

PriorityClasses: online inference (value 100000, PreemptLowerPriority), batch training (value 1000, Never). Training pods must checkpoint on SIGTERM.

Monitoring: From "Is GPU Broken?" to "Is GPU Wasted?"

Collection Chain

NVML/DCGM → dcgm-exporter (DaemonSet, :9400/metrics) → Prometheus → Alertmanager / Grafana

Operator creates DaemonSet, Service, optional ServiceMonitor. Verify scrape: curl localhost:9400/metrics | grep -c '^DCGM_'.

Key Metrics (DCGM 4.6.1)

Metric                              Unit    Meaning                          Threshold Reference
DCGM_FI_DEV_GPU_UTIL                %       SM utilization sample          Sustained <10% → waste
DCGM_FI_DEV_MEM_COPY_UTIL           %       Memory bandwidth util          High + low SM → memory bound
DCGM_FI_DEV_FB_USED                 MiB     Used VRAM                      Near total → OOM warning
DCGM_FI_DEV_GPU_TEMP                °C      Core temp                      >83° watch, >88° throttle
DCGM_FI_DEV_MEMORY_TEMP             °C      HBM temp                       >95° high risk
DCGM_FI_DEV_POWER_USAGE             W       Power draw                     Near limit → saturated
DCGM_FI_DEV_XID_ERRORS              -       Last Xid error code            Non-zero → investigate
DCGM_FI_DEV_PCIE_REPLAY_COUNTER     count   PCIe replays                   Growing → link issue
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL       count   Uncorrectable ECC              Non-zero → immediate
DCGM_FI_DEV_RETIRED_PENDING         count   Pending retired pages          Non-zero → VRAM degradation
DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL count NVLink CRC errors    Growing → interconnect fault

GPU_UTIL

is sampled (1s), may miss short tasks; reflects kernel occupancy not efficiency. Profiling metrics ( DCGM_FI_PROF_*) need explicit exporter config and add 1-3% overhead.

Prometheus Scrape Config

Use metric_relabel_configs keep-list to limit series (thousands otherwise). Example keeps FB_USED, FB_FREE, GPU_UTIL, GPU_TEMP, POWER_USAGE, XID_ERRORS, ECC_DBE, RETIRED_PENDING.

Alerting Rules

Critical: GPUXidError (value change detection), GPUMemoryECCUncorrectable, GPUAllocatableZero (capacity>0 but allocatable=0). Warning: GPURetiredPagesPending, GPUTemperatureHigh (>85°), GPUPCIeReplayGrowing. Info: GPUIdleWithPod (allocated but <5% util 30m), GPUMemoryPressure (>95% VRAM). Label names must match exporter output (e.g., Hostname, gpu, UUID).

Dashboard Panels

Capacity overview (from kube API, not DCGM).

Utilization heatmap (avg_over_time GPU_UTIL).

VRAM water level (FB_USED/(FB_USED+FB_FREE) per GPU/pod).

Hardware health (Xid, ECC, temp, power).

Denominator for utilization = allocated GPUs (API), not total GPUs.

Troubleshooting Closed Loop: 7 Production Cases

Case 1: All Pods Pending – Insufficient nvidia.com/gpu

Symptom : Scheduler events show 0 allocatable. Diagnosis : Compare physical usage (nvidia-smi) vs K8s allocatable. Metrics : allocatable=0 but DCGM FB_USED low → ledger error. Root cause : kubelet restart broke ListAndWatch; plugin pod not restarted. Evidence : kubelet logs show ListAndWatch failed: transport is closing. Fix :

kubectl rollout restart daemonset/nvidia-device-plugin-daemonset -n gpu-operator

. Verification : allocatable returns to 8, pending pods schedule. Retrospective : Add GPUAllocatableZero alert; add plugin restart to kubelet maintenance script.

Case 2: Pod Running but No GPU in Container

Symptom : nvidia-smi not found or No devices. Triage : Check /dev/nvidia* in container, env vars, actual mounts via /proc/self/mountinfo (spec volumes don't show GPU mounts). Root causes : (1) Pod missing GPU limit; (2) Device Plugin + DRA both active. Fix : Add GPU limit, recreate pod. Prevention : Admission policy (OPA/Kyverno) requiring GPU limit for CUDA images.

Case 3: Device Plugin CrashLoopBackOff

Log patterns : Driver/library version mismatch (kernel/userspace version skew → reboot or rmmod+modprobe), no NVIDIA devices found (driver not loaded), failed to create socket (stale socket → delete), Failed to register with kubelet (kubelet down or wrong socket path), MIG strategy mixed requires MIG-enabled devices (config mismatch). Fix : Reboot preferred; hot-reload modules only if no processes hold GPU. Rollback : Helm rollback or runfile uninstall + reinstall old driver.

Case 4: Xid 79 – GPU Fell Off Bus

Symptom : CUDA error: illegal memory access, NVIDIA-SMI failed to communicate. Diagnosis : dmesg shows Xid 79, GPU has fallen off the bus; lspci no longer shows device. Root cause : PCIe signal integrity, power instability, or GPU hardware fault. Fix : Cordon, taint, drain, attempt nvidia-smi --gpu-reset (usually fails), cold reboot via IPMI power cycle. Verification : dcgmi diag -r 3 (20-40 min stress test). Rollback : If Xid 79 recurs, node out of pool, hardware ticket.

Case 5: VRAM Slow Leak to OOM

Symptom : Inference service OOM after 12-36h, already allocated near total. Diagnosis : torch.cuda.memory_allocated/reserved/max_allocated; allocated growth → true leak; reserved growth → fragmentation (set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True). Root cause : App caching KV tensors in unbounded list. Mitigation : PYTORCH_CUDA_ALLOC_CONF + NVIDIA_GPU_MEMORY_LIMIT=70Gi (Operator 26.7.1+, driver 615+) to hard-limit per-container VRAM. Verification : 2h load test, stable VRAM curve.

Case 6: Driver Upgrade Breaks MIG Config

Symptom : 595→615 upgrade, nvidia-smi mig -cgi fails with NVML_ERROR_UNKNOWN, Xid 119. Root cause : Known issue cross major version without GPU reset. Fix : Reboot node (official). Alternative online: disable MIG label, pause MIG Manager, upgrade driver, nvidia-smi --gpu-reset, re-enable MIG Manager, reapply profile. Lesson : Major driver upgrades on MIG nodes require reboot in runbook.

Case 7: Node Reboot → GPU Resources Gone

Symptom : allocatable missing, plugin pod Running. Causes : (1) Kernel upgrade, DKMS not rebuilt (version mismatch), (2) Secure Boot blocks unsigned module, (3) MIG Manager disabled → MIG config lost, (4) udev/persistenced issues. Fix : dkms autoinstall + reboot; mokutil --import + MOK enroll; reapply MIG label. Prevention : Post-reboot self-check via systemd/DaemonSet startup probe.

Performance Optimization

Quantify Bottleneck First

Symptom                Evidence                    Direction
GPU low, CPU high      kubectl top pod CPU at limit  Data loading bottleneck → more workers/CPU
GPU low, CPU low       Both low                    Sync waits → check distributed comm
High util, low throughput Business metrics flat      Kernel inefficiency → memory/operator tuning
Utilization spikes     Sawtooth pattern            Small batch or lock contention

Command:

kubectl top pod -n ai-train --containers | sort -k3 -h | tail -20

.

Data Loading

PyTorch DataLoader: num_workers=8, pin_memory=True, prefetch_factor=4, persistent_workers=True. Requires enlarged /dev/shm (emptyDir Memory, 32Gi) else Bus error.

NUMA & CPU Pinning

GPU attached to specific CPU socket; cross-socket H2D bandwidth drops 20-40%. Check nvidia-smi topo -m and /sys/bus/pci/devices/.../numa_node. Enable kubelet cpuManagerPolicy: static + topologyManagerPolicy: restricted (not single-numa-node for large pods). Requires kubelet restart (high risk).

VRAM vs Batch Balance

Training priority: reduce batch + gradient accumulation → gradient checkpointing → mixed precision (bf16/fp16) → ZeRO/FSDP sharding → CPU offload (last resort). Inference: quantization (INT8/FP8/AWQ) → PagedAttention/KV cache paging → limit concurrency/seq len → multi-instance multi-GPU.

Inter-GPU Communication

Verify NVLink: nvidia-smi nvlink -s, nvidia-smi nvlink -e. Bandwidth test with nccl-tests all_reduce_perf; H100 NVLink 4.0 theoretical 450 GB/s unidirectional, expect >300 GB/s. Common pitfalls: NCCL_P2P_DISABLE=1 set, missing /dev/nvidia-peer-memory or IPC caps, IB not passed through in multi-node.

Sharing Strategy Performance

Strategy      Compute Isolation   VRAM Isolation   Context Switch Overhead   Suitable For
Exclusive     Full              Full             None                      Training, large model inference
MIG           Per slice         Full             Low                       Multi-tenant inference
Time-slicing  None              None             High (~ms per switch)     Dev, low-load inference
MPS           Proportional      Proportional     Low (shared context)      Multi-process same model

Time-slicing replicas ≤4 for non-latency-sensitive workloads.

High-Risk Operations & Rollback

Risk Classification

Operation              Impact Scope              Rollback Ease   Requirements
Label change           Single node scheduling    High            Two-person review
Restart device plugin  Node new allocations      High            Timestamp record
Containerd config change All node containers     Medium          Maintenance window + cordon
Restart kubelet        Node pod status sync      Medium          Maintenance window + drain
Driver upgrade         All GPU tasks on node     Low             Maintenance window + drain + backup
MIG reconfig           All tasks on that GPU     Medium          Maintenance window + drain GPU
Uninstall GPU Operator Cluster GPU capability    Low             Change review + full window
Switch to DRA          Cluster scheduling model  Very low        Separate cluster canary

Driver Upgrade Runbook

Phase 1 (T-3 days) : Backup node list, DS yaml, Helm values, driver versions. Validate new driver with CUDA images on test node.

Phase 2 (T-day, canary 1 node) : Cordon/drain node, disable driver deployment on other nodes via label nvidia.com/gpu.deploy.driver=false, upgrade Helm driver.version, watch driver pod logs.

Phase 3 (Verify) : Check driver version label, allocatable restored, dcgmi diag -r 2, smoke test pod.

Phase 4 (Rollout) : Re-enable driver label in 25% batches, each batch cordon/drain/label/uncordon, wait 30 min between batches monitoring P99/error rate/utilization.

Rollback : helm rollback or helm upgrade --set driver.version=old. Fallback: manual nvidia-uninstall + runfile reinstall old version + reboot.

GPU Operator Uninstall

helm uninstall

only removes CRDs/Deployments; driver and containerd config remain. Full cleanup: ensure zero GPU pods, cordon/drain all GPU nodes, uninstall chart, delete CRDs (backup ClusterPolicy first!), per-node nvidia-uninstall, purge nvidia-container-toolkit, restart containerd.

Safe Batch Operations Script

Provided gpu-node-roll.sh with set -euo pipefail, default DRY_RUN=true, explicit YES confirmation, per-node sequential execution with failure stop, dual wait (node Ready + GPU allocatable), full tee logging, SSH BatchMode/ConnectTimeout.

Production Considerations

Permissions

Cluster: RBAC Role (not ClusterRole) for cordon/drain/label on GPU node namespace.

Node: No direct SSH for nvidia-smi --gpu-reset; use approved ticket + Ansible/bastion audit.

Secrets: IPMI/BMC/registry creds via Vault/KMS, env vars only.

Device plugin SA: only get/list/watch node; never cluster-admin.

Change Windows & Canary

Change Type       Window              Canary
Driver minor      Low-traffic 2h      1 node → 25% → all
Driver major      Weekend 4h          1 node → observe 24h → all
Containerd config Low-traffic 1h      By node pool
MIG reconfig      Coordinate with workload  By GPU pool
DRA migration     New cluster         Fresh cluster, no in-place

24h observation catches memory leaks, intermittent Xids, thermal issues.

Capacity Planning

Usable GPU-hours = GPUs × 24h × available days × health rate (95-98%) × allocation rate (70-85%). Account for fragmentation (e.g., 8-GPU nodes with 1-GPU jobs leave 7 unusable for 8-GPU jobs). Example: 4×8-GPU training + 20×1-GPU inference → 9 nodes (4 dedicated training, 4 inference + growth, 1 spare). Time-slicing 4x reduces inference to 1 node but adds interference risk – business decision.

Image & Dependency Governance

Tag images with CUDA version ( pytorch:2.6-cu124); runtime CUDA ≤ driver-supported CUDA.

Never install driver in image (causes version mismatch).

Large images (3-8GB) → configure kubelet image GC: imageGCHighThresholdPercent: 85, imageGCLowThresholdPercent: 80, imageMinimumGCAge: 24h.

Private registry + pre-pull DaemonSet or registry mirror.

Logging & Audit

Must retain: kubelet logs (plugin registration, Allocate, health), kernel logs (dmesg NVRM/Xid – configure rsyslog persistent /var/log/kernel.log with 30-day rotation), change logs (who, when, node, action).

Cost Visualization

PromQL for namespace GPU count:

sum by (namespace) (kube_pod_container_resource_requests{resource="nvidia_com_gpu"})

. GPU-hours:

sum by (namespace) (avg_over_time(kube_pod_container_resource_requests{resource="nvidia_com_gpu"}[24h])) * 24

. Enables "utilization <10% → reclaim" governance.

Summary & Weekly Checklist

GPU ops difficulty lies in three-layer coupling : driver, runtime, device plugin. Any misalignment shows "GPU unusable" but root cause differs. Key takeaways:

Layer first, act later – use ctr run --gpus 0 to split driver vs K8s issues.

Watch allocatable, not capacity.

Extended resources: no overcommit, no isolation, non-compressible.

MIG = hard isolation; time-slicing = soft sharing (dev/test only).

Xid is hardware truth – non-zero must investigate, not "reboot fixed it".

Driver upgrade always needs reboot; "no-reboot" claims have preconditions.

DRA is future, not current default – canary in new clusters.

Weekly checklist script ( gpu-weekly-check.sh) checks: node resource consistency, component health, recent Xids, VRAM top10, long-pending GPU pods, driver version distribution. Top concerns: driver version divergence across cluster, pods pending >30 min (usually config errors).

Ultimately, GPU cluster stability equals "ledger matches physical reality". All monitoring, patrols, alerts serve that single verification.