GPU Scheduling, Isolation, and Resource Allocation in Kubernetes Clusters

1. Understanding the Full GPU Resource Chain

A GPU‑enabled node must pass several checks before a Pod can use a GPU: the PCIe layer must detect the device, the NVIDIA kernel driver must load, the container runtime must have the NVIDIA Container Toolkit (or equivalent CDI) configured, the Device Plugin DaemonSet must register nvidia.com/gpu (or MIG resources) with the kubelet, the node capacity and allocatable fields must list the resource, the Pod must request the resource in its limits, the scheduler must find a node that satisfies labels, taints, affinity and remaining resources, and finally kubelet must inject the device into the container where the CUDA user‑space libraries match the host driver.

PCIe layer detects GPU.

NVIDIA driver loads.

Container runtime configured with NVIDIA Toolkit/CDI.

Device Plugin DaemonSet runs and registers resources.

Node capacity / allocatable show nvidia.com/gpu or MIG entries.

Pod declares the GPU in limits (and optionally requests).

Scheduler matches node labels, taints, affinity, and resource availability.

Kubelet injects the device; container sees it via CUDA_VISIBLE_DEVICES or direct device nodes.

2. Environment Baseline

Run the following commands to verify the control‑plane version, node labels (generated by NFD/GFD), and the actual GPU capacity:

kubectl -n <namespace> version
kubectl -n <namespace> get nodes -o wide
kubectl -n <namespace> get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.product,nvidia.com/mig.capable

Typical output shows eight GPUs registered on a node:

Capacity:
  nvidia.com/gpu: 8
Allocatable:
  nvidia.com/gpu: 8

Note that this reflects registration, not current utilization.

3. Device Plugin – How Kubernetes Sees GPUs

The NVIDIA Device Plugin discovers healthy devices and registers them as extended resources. Verify the plugin DaemonSet and its logs:

kubectl -n <namespace> get daemonsets
kubectl -n <namespace> get pods -l app.kubernetes.io/name=nvidia-device-plugin -o wide
kubectl -n <namespace> logs daemonset/<plugin-name> --all-containers --tail=200

Key validation points:

Desired DaemonSet replicas equal the number of GPU nodes.

Pod Ready and no driver/NVML errors in logs.

Node allocatable contains the expected resource name and count.

Creating a minimal test Pod shows only the allocated device inside the container.

4. Minimal GPU Pod Manifest

GPU resources are declared in limits. Fractional requests are not allowed.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-check
  namespace: <namespace>
spec:
  restartPolicy: Never
  containers:
  - name: cuda
    image: <CUDA‑image‑with‑nvidia‑smi>
    command: ["bash", "-lc"]
    args:
    - |
      set -euo pipefail
      nvidia-smi -L
      nvidia-smi
    resources:
      requests:
        cpu: "1"
        memory: 2Gi
        nvidia.com/gpu: "1"
      limits:
        cpu: "2"
        memory: 4Gi
        nvidia.com/gpu: "1"

Apply with a server‑side dry‑run first:

kubectl -n <namespace> apply --server-side --dry-run=server -f pod.yaml
kubectl -n <namespace> apply -f pod.yaml

5. Scheduling Constraints – Beyond nvidia.com/gpu

When a cluster contains heterogeneous GPUs, a plain request nvidia.com/gpu: 1 may schedule onto any node. Use node labels and affinity to target a specific pool:

# Label a node
kubectl -n <namespace> label node <gpu-node> accelerator.example.com/pool=<pool-name> --overwrite

# Pod affinity example
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: accelerator.example.com/pool
            operator: In
            values:
            - <pool-name>
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        preference:
          matchExpressions:
          - key: accelerator.example.com/generation
            operator: In
            values:
            - <preferred‑GPU‑gen>

Use taints and tolerations to reserve GPU‑only nodes:

# Taint the node
kubectl -n <namespace> taint node <gpu-node> nvidia.com/gpu=present:NoSchedule

# Pod toleration
spec:
  tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: present
    effect: NoSchedule

6. Isolation Levels

Full‑card exclusive – nvidia.com/gpu: 1. Clear boundaries, predictable performance, but low utilization for small workloads.

Time‑slicing (shared) – Device Plugin can expose multiple replicas (e.g., 4) per physical GPU. No hard memory partition; all Pods share the same VRAM and fault domain. Use failRequestsGreaterThanOne: true to prevent accidental over‑request.

MIG (Multi‑Instance GPU) – Hardware partitioning into independent instances. The plugin exposes resources such as nvidia.com/mig-1g.5gb. Requires supported GPU models and driver/Operator versions.

Configuration example for time‑slicing:

version: v1
sharing:
  timeSlicing:
    renameByDefault: false
    failRequestsGreaterThanOne: true
    resources:
    - name: nvidia.com/gpu
      replicas: 4

7. MPS vs. Time‑Slicing

Choose the model based on workload characteristics:

Stable, no interference – full‑card exclusive.

Need hardware memory isolation – MIG.

Many small inference jobs, tolerate performance jitter – time‑slicing.

CUDA‑level multi‑process optimization – MPS (requires explicit enablement).

8. Namespace ResourceQuota for GPUs

Limit the total number of GPUs a namespace can request:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: <namespace>
spec:
  hard:
    requests.nvidia.com/gpu: "8"

Apply with dry‑run, then verify usage with kubectl describe resourcequota gpu-quota. Deleting a quota removes the limit but should be done after confirming alternative controls.

9. Multi‑GPU Training

A Pod requesting four GPUs must be scheduled onto a single node that has at least four allocatable GPUs. The default scheduler does not split a request across nodes; distributed training therefore requires multiple Pods, a training controller, or a gang‑scheduler that handles rendezvous, networking, and storage.

10. CPU, Memory, and NUMA Considerations

Insufficient CPU or memory can become the bottleneck for GPU workloads. For latency‑sensitive or NUMA‑aware jobs, enable the static CPU manager and single-numa-node topology manager policies in the kubelet configuration (requires node restart).

11. Diagnosing Pending Pods

Gather evidence in order:

Pod events: kubectl describe pod <pod> – look for messages like "Insufficient nvidia.com/gpu" or "untolerated taint".

Verify the Pod's resource request:

kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}'

.

Inspect node allocatable and current allocations: kubectl describe node <gpu-node> and list Pods on that node.

Check spelling of resource names; they must match exactly what the node reports.

Review admission controls (ResourceQuota, LimitRange, validating webhooks) that may modify or reject the request.

12. Pod Running but Not Seeing a GPU

Confirm the Pod actually requested a GPU limit, then exec into the container and run:

nvidia-smi -L
echo $CUDA_VISIBLE_DEVICES

Common causes include missing GPU limit, wrong RuntimeClass, failed Device Plugin registration, driver‑runtime mismatch, or the container image lacking NVIDIA libraries.

13. Monitoring GPU Resources

Kubernetes reports allocatable and Pod requests, while DCGM Exporter provides real‑time metrics such as DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_POWER_USAGE, and DCGM_FI_DEV_XID_ERRORS. Example PromQL for low utilization:

avg_over_time(DCGM_FI_DEV_GPU_UTIL[15m]) < 10

Combine scheduler data, plugin logs, and DCGM metrics to build a complete picture of GPU health.

14. Resource Fragmentation and Fair Scheduling

The default scheduler does not rearrange GPUs for future large jobs, leading to fragmentation. Mitigation strategies include:

Separate node pools for single‑card inference vs. multi‑card training.

Use taints, labels, and node affinity to protect whole‑node pools.

Implement queueing or gang‑scheduling for large jobs.

Priorities with controlled preemption, ensuring checkpointing for training workloads.

15. Upgrading Device Plugin, Driver, or Sharing Strategy

Follow a safe upgrade workflow:

Identify affected node pools, GPU models, current workloads, and version matrix.

Backup DaemonSet YAML, node status, and Helm/Operator values.

Run server‑side dry‑run and diff against the new manifest.

Cordon a single node, perform the upgrade, and validate with minimal test Pods (full‑card, time‑slicing, MIG).

Uncordon the node, repeat on other nodes, and monitor allocatable and plugin logs.

If problems arise, roll back to the previous image digest and configuration, then repeat validation.

16. Daily Inspection Checklist

# GPU capacity per node
kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

# Pending GPU Pods
kubectl get pods --field-selector=status.phase=Pending
kubectl get events --sort-by=.lastTimestamp | tail -n 100
kubectl get resourcequota

# Host‑level sanity checks
nvidia-smi -L
nvidia-smi --query-gpu=uuid,name,temperature.gpu,power.draw,memory.used,memory.total,utilization.gpu --format=csv

Watch for sudden changes in Device Plugin readiness, node capacity, Xid errors, prolonged high memory usage, or increasing pending times.

17. Common Misconceptions

Seeing nvidia-smi succeed on the host does not guarantee Kubernetes scheduling.

Low GPU utilization does not free the allocated card for another Pod unless a shared model is used.

Time‑slicing replicas are not hard memory partitions.

Tolerations allow scheduling onto a tainted node but do not allocate a GPU.

Eight free cards spread across three nodes cannot satisfy a single‑Pod request for eight GPUs.

Deleting a pending Pod without fixing the underlying cause will cause the controller to recreate it.

Conclusion

Kubernetes manages GPUs through the Device Plugin framework, which registers only the quantity ( nvidia.com/gpu) as an extended resource. Full‑card exclusive allocation provides clear boundaries; MIG offers hardware‑level memory isolation; time‑slicing and MPS enable sharing with trade‑offs. Use node labels, taints, ResourceQuota, and careful monitoring to build reliable GPU pools, and always perform upgrades with backups, dry‑runs, single‑node gray‑scale, and explicit rollback steps.