Mastering GPU Scheduling, Isolation, and Resource Allocation in Kubernetes Clusters

Understanding the Full GPU Resource Chain

For a GPU pod to run, the following steps must succeed:

PCIe layer on the host detects the GPU.

NVIDIA kernel driver loads and the host can query the device.

The container runtime is configured with NVIDIA Container Toolkit or equivalent CDI support.

The Device Plugin DaemonSet runs on GPU nodes and registers nvidia.com/gpu (or MIG resources) with the kubelet.

Node capacity and allocatable report the GPU resources.

Pod limits declare the required GPU count.

The scheduler selects a node whose labels, taints, affinity, and remaining resources satisfy the request.

Kubelet injects the allocated device into the container.

The container’s CUDA user‑space libraries match the host driver and the application selects the device correctly.

When troubleshooting, verify each layer instead of only checking nvidia-smi inside the pod.

Environment Inventory – Building a Version and Hardware Baseline

Run the following commands to capture the cluster state (replace <namespace> and <GPU节点名> with real values):

kubectl -n <namespace> version
kubectl -n <namespace> get nodes -o wide
kubectl -n <namespace> get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.product,nvidia.com/mig.capable

Node labels are usually generated by Node Feature Discovery (NFD) and GPU Feature Discovery (GFD). Even without these labels, the GPU may still be usable; continue checking the node’s status.capacity and status.allocatable fields.

kubectl -n <namespace> get node <GPU节点名> \
  -o jsonpath='{.status.capacity}
{.status.allocatable}
'
kubectl -n <namespace> describe node <GPU节点名>

Example output shows 8 GPUs registered:

Capacity:
  nvidia.com/gpu:  8
Allocatable:
  nvidia.com/gpu:  8

Note that this reflects the number of allocatable GPU devices, not current utilization.

Device Plugin – How Kubernetes Sees GPUs

The NVIDIA Device Plugin registers GPUs as the extended resource nvidia.com/gpu. The scheduler only sees the count; actual device IDs are chosen during the kubelet allocation phase.

kubectl -n <namespace> get daemonsets
kubectl -n <namespace> get pods -o wide -l app.kubernetes.io/name=nvidia-device-plugin

Validate the DaemonSet:

Desired replicas match the number of GPU nodes.

Pod is Ready and logs show no driver or NVML errors.

Node allocatable contains the expected resource name and quantity.

A minimal test pod sees only the allocated device.

Basic GPU Pod Manifest

GPU requests must be integer values; fractional requests are not allowed.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-check
  namespace: <namespace>
spec:
  restartPolicy: Never
  containers:
  - name: cuda
    image: <CUDA image with nvidia-smi>
    command: ["bash", "-lc"]
    args: |
      set -euo pipefail
      nvidia-smi -L
      nvidia-smi
    resources:
      requests:
        cpu: "1"
        memory: 2Gi
        nvidia.com/gpu: "1"
      limits:
        cpu: "2"
        memory: 4Gi
        nvidia.com/gpu: "1"

Apply with a server‑side dry‑run first:

kubectl -n <namespace> apply --server-side --dry-run=server -f pod.yaml
kubectl -n <namespace> apply -f pod.yaml
kubectl -n <namespace> get pod gpu-check -o wide
kubectl -n <namespace> describe pod gpu-check
kubectl -n <namespace> logs gpu-check

Delete the test pod after verification:

kubectl -n <namespace> delete pod gpu-check --dry-run=server
kubectl -n <namespace> delete pod gpu-check

Scheduling Constraints – Beyond Simple nvidia.com/gpu

When a cluster contains heterogeneous GPUs, use node labels and pod affinity to steer workloads to the appropriate pool.

# Label a node with a stable pool name
kubectl -n <namespace> label node <GPU节点名> \
  accelerator.example.com/pool=<pool-name> --overwrite --dry-run=server -o yaml
kubectl -n <namespace> label node <GPU节点名> \
  accelerator.example.com/pool=<pool-name> --overwrite

# Pod affinity example
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: accelerator.example.com/pool
            operator: In
            values:
            - <pool-name>
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        preference:
          matchExpressions:
          - key: accelerator.example.com/generation
            operator: In
            values:
            - <preferred‑GPU‑generation>

Hard constraints ( required) cause the pod to stay Pending if not satisfied; soft preferences ( preferred) only affect scoring.

GPU‑specific nodes can be protected with taints:

# Add a NoSchedule taint
kubectl -n <namespace> taint node <GPU节点名> \
  nvidia.com/gpu=present:NoSchedule --dry-run=server -o yaml
kubectl -n <namespace> taint node <GPU节点名> \
  nvidia.com/gpu=present:NoSchedule

# Pod toleration
spec:
  tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: present
    effect: NoSchedule

Remember that tolerations only allow scheduling onto the node; the pod must still request the GPU resource.

Three Levels of GPU Isolation

1. Full‑Card Exclusive

Requesting nvidia.com/gpu: 1 allocates an entire physical GPU. This gives clear resource boundaries and predictable performance, but low utilization for small workloads.

Avoid privileged containers or manual /dev/nvidia* mounts that can bypass the plugin.

2. Time‑Slicing (Shared GPUs)

The Device Plugin can expose a GPU as multiple virtual replicas. Example config (values depend on the installed plugin version):

version: v1
sharing:
  timeSlicing:
    renameByDefault: false
    failRequestsGreaterThanOne: true
    resources:
    - name: nvidia.com/gpu
      replicas: 4

Key limits:

Replicas are not real GPUs and do not guarantee fixed compute share.

All replicas share the same memory and fault domain.

One process exhausting memory can affect all co‑located pods.

The scheduler only sees the replica count, not real utilization.

3. MIG (Hardware Partitioning)

Supported on MIG‑capable GPUs, each MIG profile appears as a distinct extended resource such as nvidia.com/mig-1g.5gb. Query the node to discover actual names:

kubectl -n <namespace> get node <GPU节点名> -o json \ 
  | jq '.status.capacity | with_entries(select(.key | startswith("nvidia.com/")))'

Pod request example:

resources:
  requests:
    nvidia.com/mig-1g.5gb: "1"
  limits:
    nvidia.com/mig-1g.5gb: "1"

MIG provides hard memory and compute isolation, but profile combinations are limited by hardware and re‑configuring MIG on a node affects all workloads.

MPS vs. Time‑Slicing vs. MIG

Stable, no interference – use full‑card exclusive.

Need hardware memory partitioning – use MIG.

Small concurrent tasks tolerating performance jitter – use time‑slicing.

Optimized CUDA multi‑process sharing – enable MPS where supported.

Never enable multiple sharing modes simultaneously on the same node without isolating them into separate pools.

Namespace Quotas – Controlling GPU Consumption per Team

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: <namespace>
spec:
  hard:
    requests.nvidia.com/gpu: "8"

Apply with a dry‑run first, then verify the quota:

kubectl -n <namespace> apply --server-side --dry-run=server -f quota.yaml
kubectl -n <namespace> apply -f quota.yaml
kubectl -n <namespace> describe resourcequota gpu-quota

When using MIG or time‑slicing, quota must reference the actual extended resource names.

Multi‑GPU Training – Node‑Level Constraints

A pod requesting 4 GPUs requires a single node with at least 4 allocatable GPUs; the scheduler does not split the request across nodes. Example request:

resources:
  requests:
    nvidia.com/gpu: "4"
  limits:
    nvidia.com/gpu: "4"

Ensure the node’s topology (NVLink, NVSwitch) matches the workload’s communication pattern and set CUDA_VISIBLE_DEVICES inside the container accordingly.

CPU, Memory, and NUMA – Hidden Bottlenecks

Insufficient CPU or memory can become the real bottleneck. Use static CPU manager and single‑NUMA topology manager for latency‑sensitive workloads:

kubelet:
  cpuManagerPolicy: static
  topologyManagerPolicy: single-numa-node

Changing these settings requires a kubelet restart and affects all pods on the node.

Evidence‑Based Troubleshooting of Pending Pods

Inspect pod events and describe output for messages such as "Insufficient nvidia.com/gpu" or "untolerated taint".

Verify the pod’s actual resource request with kubectl get pod -o jsonpath.

Check node allocatable and current pod allocations.

Confirm the resource name spelling matches node capacity (e.g., nvidia.com/gpu or MIG names).

Review scheduler and admission policies (ResourceQuota, LimitRange, ValidatingAdmissionPolicy, webhooks).

Example event output:

0/6 nodes are available: 2 Insufficient nvidia.com/gpu, 2 node(s) had untolerated taint, 2 node(s) didn't match Pod's node affinity/selector.

Pods Running but Not Seeing GPUs

Confirm the pod actually requested a GPU:

kubectl -n <namespace> get pod <pod> -o jsonpath='{.spec.containers[*].resources.limits}
'

Enter the container and check:

kubectl -n <namespace> exec -it <pod> -c <container> -- nvidia-smi -L
kubectl -n <namespace> exec -it <pod> -c <container> -- sh -c 'printf "%s
" "${CUDA_VISIBLE_DEVICES:-<未设置>}"'

Common root causes include missing GPU limit, wrong RuntimeClass, plugin registration failure, image lacking nvidia-smi, driver‑library incompatibility, or stale MIG/time‑slicing configuration.

Monitoring – Correlating Scheduler State and Device Metrics

Kubernetes allocatable shows how many GPUs are assigned, while tools like NVIDIA DCGM Exporter expose real‑time utilization. DCGM_FI_DEV_GPU_UTIL – GPU utilization. DCGM_FI_DEV_MEM_COPY_UTIL – memory bandwidth usage. DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE – memory used / free. DCGM_FI_DEV_GPU_TEMP – temperature. DCGM_FI_DEV_POWER_USAGE – power draw. DCGM_FI_DEV_XID_ERRORS – Xid error codes.

Sample PromQL queries:

avg_over_time(DCGM_FI_DEV_GPU_UTIL[15m]) < 10
max_over_time(DCGM_FI_DEV_FB_USED[10m])
DCGM_FI_DEV_XID_ERRORS > 0

Dashboards should label metrics by namespace, workload, pod, node, GPU UUID/MIG UUID, and resource pool to avoid hiding per‑GPU contention.

Resource Fragmentation and Fair Scheduling

The default scheduler places pods based on current free resources, which can leave many single‑GPU tasks scattered across nodes, preventing a large multi‑GPU job from finding a node with enough free cards. Mitigations include:

Separate node pools for inference (single‑GPU) and training (multi‑GPU).

Use taints, labels, and node affinity to protect whole‑node pools.

Implement queueing or gang‑scheduling for large jobs.

Prioritize critical workloads and control preemption carefully.

Ensure checkpointing for training jobs before any preemptive eviction.

Pre‑emptive pod deletion without proper PDBs or checkpointing can cause hours of lost training progress.

Standard Procedure for Upgrading Device Plugin, Drivers, or Sharing Strategies

Identify the impact scope – list target node pools, GPU models, current workloads, MIG/time‑slicing config, runtime versions, and driver versions.

Backup current DaemonSet, node YAML, and list of GPU‑using pods.

kubectl -n <namespace> get daemonset <device‑plugin> -o yaml > plugin-backup.yaml
kubectl -n <namespace> get node <GPU节点名> -o yaml > node-backup.yaml
kubectl -n <namespace> get pods --all-namespaces --field-selector spec.nodeName=<GPU节点名> -o wide

Dry‑run the new manifest and diff:

kubectl -n <namespace> diff -f new-plugin.yaml
kubectl -n <namespace> apply --server-side --dry-run=server -f new-plugin.yaml

Gray‑scale on a single node: cordon the node, then apply the upgrade.

kubectl -n <namespace> cordon <GPU节点名>
# upgrade steps here
kubectl -n <namespace> uncordon <GPU节点名>

Validate post‑upgrade:

kubectl -n <namespace> get node <GPU节点名> -o jsonpath='{.status.allocatable}
'
kubectl -n <namespace> get pods -o wide
kubectl -n <namespace> logs daemonset/<device‑plugin> --all-containers --tail=200
# run minimal single‑GPU, multi‑GPU, MIG test pods

Rollback if needed – restore the original DaemonSet image digest, Helm values, or Operator CR, then repeat the validation steps.

Daily Inspection Checklist

# GPU capacity per node
kubectl -n <namespace> get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
# Pending pods
kubectl -n <namespace> get pods --field-selector=status.phase=Pending
# Recent events
kubectl -n <namespace> get events --sort-by='.lastTimestamp' | tail -n 100
# Namespace quotas
kubectl -n <namespace> get resourcequota
# Host‑level checks (run on the node)
nvidia-smi -L
nvidia-smi --query-gpu=uuid,name,temperature.gpu,power.draw,memory.used,memory.total,utilization.gpu --format=csv

Watch for sudden changes in Device Plugin readiness, node GPU capacity, Xid errors, high memory usage, growing pending times, or mixed hardware in a single pool.

Common Misconceptions

Seeing nvidia-smi work on the host does not guarantee Kubernetes can schedule GPUs.

Low GPU utilization does not mean another pod can be scheduled; the scheduler only sees allocated counts.

Time‑slicing replicas do not provide hard memory partitions.

Tolerating a GPU taint does not allocate a GPU; the pod must still request the resource.

Eight free GPUs spread across multiple nodes cannot satisfy a single pod requesting eight GPUs.

Deleting a pending pod without fixing the underlying cause will cause the controller to recreate it.

Conclusion

Kubernetes manages GPUs through a three‑layer model: the Device Plugin discovers and registers devices, the scheduler matches resource requests with node capacity, and the underlying hardware and runtime enforce isolation. Use full‑card exclusive as a stable baseline, then adopt MIG, time‑slicing, or MPS as needed. Protect resource pools with labels, taints, and namespace quotas, correlate scheduler state with DCGM metrics, and perform every change with backup, dry‑run, single‑node gray‑scale, thorough validation, and a clear rollback plan to keep GPU utilization high without sacrificing reliability.