How to Schedule, Isolate, and Allocate GPUs in a Kubernetes Cluster

GPU resource flow from node to container

PCIe on the host detects the NVIDIA GPU.

The NVIDIA kernel driver loads; nvidia-smi can query the device.

The container runtime is configured with the NVIDIA Container Toolkit or an equivalent CDI implementation.

The NVIDIA Device Plugin runs as a DaemonSet on GPU nodes and registers nvidia.com/gpu (or MIG resources) with the kubelet.

The node capacity and allocatable fields expose those extended resources.

A Pod declares the required GPU in resources.limits (and optionally resources.requests).

The scheduler selects a node whose labels, taints, affinity and remaining resources satisfy the request.

Kubelet and the runtime inject the allocated device files into the container.

The container’s CUDA user‑space must be compatible with the host driver; the application selects the device via CUDA_VISIBLE_DEVICES or device UUID.

Environment inventory

Kubernetes control‑plane view

kubectl -n <namespace> version
kubectl -n <namespace> get nodes -o wide
kubectl -n <namespace> get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.product,nvidia.com/mig.capable

Node labels are usually added by Node Feature Discovery (NFD) and GPU Feature Discovery (GFD). Missing labels do not mean the GPU is unavailable; check the node capacity and allocatable fields.

Host‑side checks

lspci -nn | grep -i nvidia
nvidia-smi
nvidia-smi -L
lsmod | grep '^nvidia'

Kubelet and container‑runtime logs

sudo journalctl -u kubelet --since '<start‑time>' --no-pager | grep -Ei 'device.?plugin|nvidia|cdi|allocate|registration'
sudo crictl info
sudo crictl ps -a | grep -Ei 'nvidia|device-plugin'

If the cluster uses containerd, also verify its configuration before restarting. Restarting containerd or kubelet may affect all workloads on the node; cordon the node and evaluate impact first.

Device Plugin verification

The plugin discovers healthy GPUs and registers them as extended resources. The scheduler only sees the quantity; actual device IDs are chosen during the kubelet allocation phase.

Check DaemonSet and Pods

kubectl -n <namespace> get daemonsets
kubectl -n <namespace> get pods -l app.kubernetes.io/name=nvidia-device-plugin -o wide

Validate the following:

Desired DaemonSet count matches the number of GPU nodes.

Pod is Ready and logs show no driver or NVML initialization errors.

Node allocatable contains the expected resource name and count.

A minimal test Pod sees only the allocated device.

Basic GPU Pod specification

Extended resources must be declared in limits. GPU requests are integer values; fractional requests such as 0.5 are rejected. If only limits are set, the scheduler treats them as the request value.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-check
  namespace: <namespace>
spec:
  restartPolicy: Never
  containers:
  - name: cuda
    image: <CUDA‑image‑with‑nvidia‑smi>
    command: ["bash", "-lc"]
    args: |
      set -euo pipefail
      nvidia-smi -L
      nvidia-smi
    resources:
      requests:
        cpu: "1"
        memory: 2Gi
        nvidia.com/gpu: "1"
      limits:
        cpu: "2"
        memory: 4Gi
        nvidia.com/gpu: "1"

Apply with a server‑side dry‑run first, then create the Pod and verify the device list inside the container.

Scheduling constraints – beyond nvidia.com/gpu

Clusters often contain heterogeneous GPU models, memory sizes and interconnects. Use node labels to express hardware pools and apply nodeAffinity in the Pod.

Label a node

kubectl -n <namespace> label node <gpu-node> accelerator.example.com/pool=<pool-name> --overwrite --dry-run=server -o yaml
kubectl -n <namespace> label node <gpu-node> accelerator.example.com/pool=<pool-name> --overwrite

Pod affinity and preference

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: accelerator.example.com/pool
            operator: In
            values:
            - <pool-name>
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        preference:
          matchExpressions:
          - key: accelerator.example.com/generation
            operator: In
            values:
            - <preferred‑GPU‑generation>

Required constraints affect scheduling; preferred constraints only influence scoring.

GPU‑only nodes with taints

kubectl -n <namespace> taint node <gpu-node> nvidia.com/gpu=present:NoSchedule

Pods that need a GPU add a matching toleration:

spec:
  tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: present
    effect: NoSchedule

GPU isolation levels

1. Full‑card exclusive

Declare nvidia.com/gpu: 1 to allocate an entire GPU. Benefits: clear resource boundaries, predictable performance, limited fault impact. Drawbacks: low utilization for small workloads.

Avoid privileged containers or manual /dev/nvidia* mounts, which can bypass the isolation.

2. Time‑slicing (shared)

The NVIDIA Device Plugin can expose a physical GPU as multiple scheduling replicas. Example configuration (YAML) for four replicas:

version: v1
sharing:
  timeSlicing:
    renameByDefault: false
    failRequestsGreaterThanOne: true
resources:
- name: nvidia.com/gpu
  replicas: 4

Key points:

Replicas are not fixed compute slices; they share the same memory and fault domain.

One process exhausting memory can affect all co‑located workloads.

Scheduler sees only the replica count, not real‑time utilization.

Setting failRequestsGreaterThanOne: true prevents accidental over‑request.

3. MIG (hardware partitioning)

On MIG‑capable GPUs, each physical GPU can be split into multiple hardware instances. The Device Plugin exposes resources such as nvidia.com/mig-<profile>. Query the actual resource names on a node:

kubectl -n <namespace> get node <gpu-node> -o json \| jq '.status.capacity | with_entries(select(.key | startswith("nvidia.com/")))'

Pod request must use the exact MIG resource name:

resources:
  requests:
    nvidia.com/mig-<profile>: "1"
  limits:
    nvidia.com/mig-<profile>: "1"

MIG provides independent compute, memory and fault isolation, but profile combinations are limited by the hardware.

MPS vs. time‑slicing

CUDA Multi‑Process Service (MPS) improves concurrency for supported workloads but does not give strong isolation. Choose a model based on the following trade‑offs:

Stable, minimal interference – full‑card exclusive.

Hardware memory partitioning – MIG.

High concurrency with tolerable jitter – time‑slicing.

Verified CUDA multi‑process optimization – MPS.

Do not enable multiple sharing modes on the same node without explicit isolation.

Namespace ResourceQuota for GPU control

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: <namespace>
spec:
  hard:
    requests.nvidia.com/gpu: "8"

Apply with a dry‑run, then enforce. Quotas limit the total requested GPUs in the namespace but do not enforce utilization or fairness; additional queueing or batch schedulers are needed for true fair‑share.

Multi‑GPU training – quantity ≠ topology

A Pod requesting nvidia.com/gpu: 4 must run on a single node with at least four allocatable GPUs. The scheduler will not split the request across nodes. For distributed training, use multiple Pods or a training controller that handles rendezvous, networking and storage.

Check node topology (NVLink/NVSwitch) and set CUDA_VISIBLE_DEVICES appropriately inside containers; device UUIDs may differ from host indices.

CPU, memory and NUMA impact

Insufficient CPU or memory can become a bottleneck for data loading and communication. Over‑provisioning CPU/memory makes scheduling harder. For latency‑sensitive workloads, enable the kubelet static CPU manager and the single-numa-node topology manager:

kubelet:
  cpuManagerPolicy: static
  topologyManagerPolicy: single-numa-node

These are node‑level settings; changing them requires a kubelet restart and may affect all Pods on the node.

Troubleshooting pending Pods

Inspect Pod events and kubectl describe pod output.

Verify the Pod’s resources.limits and resources.requests for the correct GPU resource name.

Check node allocatable and current allocations.

Confirm the exact resource name spelling matches the node capacity (e.g., nvidia.com/gpu, nvidia.com/mig‑...).

Review ResourceQuota, LimitRange, admission policies or custom schedulers that may reject the request.

Typical event message:

0/6 nodes are available: 2 Insufficient nvidia.com/gpu, 2 node(s) had untolerated taint, 2 node(s) didn't match Pod's node affinity/selector.

Address each cause before expanding the cluster.

Pod runs but cannot see GPU

Confirm the Pod actually requested a GPU limit.

Exec into the container and run nvidia-smi -L and echo $CUDA_VISIBLE_DEVICES.

Check Device Plugin logs, kubelet allocation messages and runtime configuration.

Common reasons include missing GPU limit, wrong RuntimeClass, driver‑library mismatch, privileged containers that bypass injection, or stale MIG/time‑slicing configuration.

Monitoring GPU resources and utilization

Kubernetes allocatable and Pod requests show scheduled capacity. Real‑time usage comes from exporters such as NVIDIA DCGM Exporter. Important metrics: DCGM_FI_DEV_GPU_UTIL – GPU utilization. DCGM_FI_DEV_MEM_COPY_UTIL – memory bandwidth utilization. DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE – VRAM usage. DCGM_FI_DEV_GPU_TEMP – temperature. DCGM_FI_DEV_POWER_USAGE – power draw. DCGM_FI_DEV_XID_ERRORS – Xid error codes.

Example PromQL for low utilization (average < 10% over 15 min):

avg_over_time(DCGM_FI_DEV_GPU_UTIL[15m]) < 10

Combine the metric with labels for namespace, workload, node and GPU UUID to build a complete dashboard.

Resource fragmentation and fair scheduling

The default scheduler places Pods based on current free resources and does not reorganize GPUs for future large jobs. To avoid fragmentation:

Separate single‑card inference and multi‑card training into different node pools.

Use taints, labels and node affinity to protect whole‑node GPU pools.

Implement queueing and admission control for large jobs.

Consider gang‑scheduling or batch schedulers for distributed training.

Use priority and preemption carefully, ensuring PodDisruptionBudget and checkpointing are in place.

Preemption is not cost‑free; it can lose training progress or disrupt services.

Standard upgrade procedure for Device Plugin, driver or sharing strategy

Identify the affected node pool, GPU models, current workloads, MIG/time‑slicing config, runtime class and driver version.

Backup the Device Plugin DaemonSet YAML, node status and list of workloads on the target nodes.

Run a server‑side dry‑run and kubectl diff against the new manifest.

Cordon a single node, drain if possible, then apply the upgrade on that node.

Validate node allocatable, plugin logs and run minimal test Pods for exclusive, shared and MIG scenarios.

Uncordon the node after successful verification.

If problems arise, roll back by re‑applying the original DaemonSet image digest and restoring the previous kubelet configuration.

Daily inspection checklist

kubectl -n <namespace> get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
kubectl -n <namespace> get pods --field-selector=status.phase=Pending
kubectl -n <namespace> get events --sort-by='.lastTimestamp' | tail -n 100
kubectl -n <namespace> get resourcequota
nvidia-smi -L
nvidia-smi --query-gpu=uuid,name,temperature.gpu,power.draw,memory.used,memory.total,utilization.gpu --format=csv

Watch for decreasing Device Plugin Ready counts, sudden changes in node GPU capacity, Xid errors, prolonged high memory usage, increasing pending times, or mixed hardware pools.