How to Use Kubernetes Node Affinity to Schedule Large Models on Specific GPU Nodes

Turning Requirements into Scheduling Constraints

Distinguish required versus preferred constraints. Required constraints (hard) include GPU model, minimum memory, and dedicated node pool. Preferred constraints (soft) include preferred availability zone and power‑profile. Replica distribution is handled by topology spread constraints. Verify cluster version, resources and permissions before proceeding.

kubectl -n NAMESPACE version --client
kubectl -n NAMESPACE version
kubectl -n NAMESPACE auth can-i list nodes
kubectl -n NAMESPACE api-resources | grep -E '^(nodes|pods|deployments)\b'

If the account cannot read nodes, request minimal read‑only permissions. All fields and flag behavior depend on the server version and --help output.

Inventory Node Facts

Inspect GPU capacity, allocatable amount, zone and taints:

kubectl get nodes -n NAMESPACE \
  -o custom-columns='NAME:.metadata.name,GPU-CAP:.status.capacity.nvidia.com/gpu,GPU-ALLOC:.status.allocatable.nvidia.com/gpu,ZONE:.metadata.labels.topology.kubernetes.io/zone,TAINTS:.spec.taints'

capacity

shows discovered resources; allocatable shows what pods can request. An empty field usually means the device plugin has not registered the extended resource, not an affinity failure.

Verify the GPU device plugin:

kubectl get daemonsets -n GPU_PLUGIN_NAMESPACE -o wide
kubectl get pods -n GPU_PLUGIN_NAMESPACE -o wide
kubectl logs -n GPU_PLUGIN_NAMESPACE DEVICE_PLUGIN_POD --all-containers=true --since=30m

If NVML initialization fails, the driver or runtime is unavailable; adding affinity alone will only keep the pod pending.

The following script aggregates GPU requests of all pods in a namespace. It depends on jq and reflects scheduling requests, not instantaneous memory usage:

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="NAMESPACE"
kubectl get pods -n "$NAMESPACE" -o json | jq -r '
  .items[]
  | select(.spec.nodeName != null)
  | [.spec.nodeName, .metadata.name,
     ([.spec.containers[].resources.requests["nvidia.com/gpu"] // "0"]
      | map(tonumber) | add)]
  | @tsv'
  | sort -k1,1

When using MIG, time‑slicing, or custom resource names, replace the resource key accordingly.

Establish Manageable Labels

Labels are scheduling inputs, not hardware facts. GPU Feature Discovery can generate labels automatically, but the key names must match the actual node.

kubectl get nodes -n NAMESPACE --show-labels | grep -E 'gpu|nvidia|accelerator|llm-pool'
kubectl get node NODE_NAME -n NAMESPACE -o json | jq '.metadata.labels'

Online node pools can use organization‑domain labels protected by NodeRestriction. Save the old value before overwriting; --overwrite replaces a key with the same name.

NODE="NODE_NAME"
kubectl get node "$NODE" -n NAMESPACE -o jsonpath='{.metadata.labels}' > "${NODE}.labels.before.txt"
kubectl label node "$NODE" -n NAMESPACE \
  ops.example.com.node-restriction.kubernetes.io/llm-pool=online \
  accelerator.example.com/gpu-family=a100 \
  accelerator.example.com/gpu-memory=80gb \
  --overwrite
kubectl get node "$NODE" -n NAMESPACE --show-labels

Label changes do not restart the node or evict existing pods, but they affect subsequent scheduling. To delete a label, use a trailing - on the key:

kubectl label node NODE_NAME -n NAMESPACE \
  ops.example.com.node-restriction.kubernetes.io/llm-pool- \
  accelerator.example.com/gpu-family- \
  accelerator.example.com/gpu-memory-

Before deletion, search for workloads that depend on those labels; after deletion, existing pods may keep running while new replicas cannot be scheduled.

Taint‑Protected Dedicated Nodes

Affinity decides where a pod goes; taints prevent ineligible pods from entering. Check current node load before adding a NoSchedule taint. Adding the taint does not evict running pods but changes future scheduling.

kubectl get pods -n NAMESPACE --field-selector spec.nodeName=NODE_NAME -o wide
kubectl taint node NODE_NAME -n NAMESPACE dedicated=llm-online:NoSchedule
kubectl describe node NODE_NAME -n NAMESPACE | sed -n '/Taints:/,/Unschedulable:/p'

Rollback uses the exact same key, value and effect:

kubectl taint node NODE_NAME -n NAMESPACE dedicated=llm-online:NoSchedule-
kubectl get node NODE_NAME -n NAMESPACE -o jsonpath='{.spec.taints}'

Hard‑Affinity Deployment

requiredDuringSchedulingIgnoredDuringExecution

means the scheduler must satisfy the rule; label changes after scheduling do not automatically evict the pod. The following manifest requires the online pool, A100 GPU family, and 80 GB memory simultaneously:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  namespace: NAMESPACE
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 0
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      tolerations:
      - key: dedicated
        operator: Equal
        value: llm-online
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: ops.example.com.node-restriction.kubernetes.io/llm-pool
                operator: In
                values: [online]
              - key: accelerator.example.com/gpu-family
                operator: In
                values: [a100]
              - key: accelerator.example.com/gpu-memory
                operator: In
                values: [80gb]
      containers:
      - name: server
        image: IMAGE_ADDRESS
        args: ["--model", "MODEL_DIR"]
        ports:
        - name: http
          containerPort: 8000
        resources:
          requests:
            cpu: "4"
            memory: 32Gi
            nvidia.com/gpu: "1"
          limits:
            cpu: "8"
            memory: 48Gi
            nvidia.com/gpu: "1"
        readinessProbe:
          httpGet:
            path: /health
            port: http
          periodSeconds: 10
          failureThreshold: 6

Expressions inside a single nodeSelectorTerm are AND‑ed; multiple terms are OR‑ed. Splitting the three conditions into separate terms unintentionally relaxes the constraint. Run a server‑side dry‑run and diff before applying:

kubectl apply -n NAMESPACE -f llm-inference.yaml \
  --dry-run=server -o yaml > /tmp/llm-inference.rendered.yaml
kubectl diff -n NAMESPACE -f llm-inference.yaml || true

If kubectl diff reports differences, manually examine image, resources, affinity and replica changes. Save the current live object before deployment:

kubectl get deployment llm-inference -n NAMESPACE -o yaml > llm-inference.before.yaml
kubectl apply -n NAMESPACE -f llm-inference.yaml
kubectl rollout status deployment/llm-inference -n NAMESPACE --timeout=10m

When GPUs have no redundancy, maxSurge: 0 prevents a new replica from occupying an extra card, but it will first reduce an old replica. If capacity cannot drop, pre‑reserve GPUs before adjusting the rolling‑update strategy.

Soft Preferences and Replica Distribution

Accept other zones but prefer zone-a using soft affinity. Weights 1‑100 are scoring values, not probabilities:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: accelerator.example.com/gpu-family
          operator: In
          values: [a100]
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 80
      preference:
        matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values: [zone-a]
    - weight: 20
      preference:
        matchExpressions:
        - key: power-profile
          operator: In
          values: [balanced]

Replica spreading uses topology spread constraints:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: llm-inference
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: ScheduleAnyway
  labelSelector:
    matchLabels:
      app: llm-inference

If candidate nodes are insufficient, DoNotSchedule leaves replicas pending. Ensure candidate nodes have the required topology labels and that the labelSelector matches the pod itself.

Validate Scheduling and Devices

Map pods, nodes, GPU requests and labels:

kubectl get pods -n NAMESPACE -l app=llm-inference \
  -o custom-columns='POD:.metadata.name,PHASE:.status.phase,NODE:.spec.nodeName,GPU:.spec.containers[*].resources.requests.nvidia.com/gpu'
kubectl get nodes -n NAMESPACE -l accelerator.example.com/gpu-family=a100 \
  -o custom-columns='NODE:.metadata.name,POOL:.metadata.labels.ops.example.com.node-restriction.kubernetes.io/llm-pool,MEM:.metadata.labels.accelerator.example.com/gpu-memory'

Each pod’s node must belong to the candidate set, GPU request must be non‑empty, and replica spreading must satisfy constraints. Running does not guarantee the model loaded successfully. If the container image includes nvidia‑smi, run it inside the container to verify the device:

POD=$(kubectl get pods -n NAMESPACE -l app=llm-inference -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n NAMESPACE "$POD" -- nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader

Mismatched model name indicates label or device‑mapping issues; invisible devices point to runtime, device‑plugin or request problems. If nvidia‑smi is unavailable, use the application’s CUDA detection API without modifying the production image.

Check probes, logs and events for further clues:

kubectl describe pod -n NAMESPACE POD_NAME
kubectl logs -n NAMESPACE POD_NAME -c server --since=20m
kubectl get events -n NAMESPACE --sort-by='.metadata.creationTimestamp' | tail -n 50

Only when the event shows didn’t match Pod’s node affinity can you conclude that candidate labels do not satisfy the requirement; a CUDA OOM in logs is a runtime memory issue, not a scheduler problem.

Evidence‑Based Pending Investigation

The following multi‑line script collects pod conditions, affinity, tolerations, GPU requests, events and candidate nodes. It is read‑only and suitable as a ticket attachment:

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="NAMESPACE"
POD="POD_NAME"

kubectl get pod "$POD" -n "$NAMESPACE" -o wide
kubectl get pod "$POD" -n "$NAMESPACE" -o json | jq '{
  conditions: .status.conditions,
  affinity: .spec.affinity,
  tolerations: .spec.tolerations,
  gpuRequests: [.spec.containers[].resources.requests["nvidia.com/gpu"]]
}'
kubectl get events -n "$NAMESPACE" \
  --field-selector "involvedObject.kind=Pod,involvedObject.name=$POD" \
  --sort-by='.metadata.creationTimestamp'
kubectl get nodes -n "$NAMESPACE" -l accelerator.example.com/gpu-family=a100 --show-labels

Evidence: didn’t match Pod’s node affinity — Judgment: hard condition has no candidates — Action: compare expression, label key and value.

Evidence: had untolerated taint — Judgment: missing matching toleration — Action: add the required toleration after confirming workload eligibility.

Evidence: Insufficient nvidia.com/gpu — Judgment: candidate node GPUs already requested — Action: scale up, free legitimate workloads, or reduce replica count.

Evidence: Too many pods — Judgment: node pod limit reached — Action: check CNI, node limits and stray pods.

Evidence: CUDA OOM after scheduling — Judgment: memory requirement exceeds GPU capacity — Action: adjust quantization, parallelism or select a larger GPU.

Strategy Drift Inspection

Hard affinity does not automatically evict pods when labels change at runtime. The script below detects label‑to‑node mismatches without performing risky deletions:

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="NAMESPACE"
SELECTOR="app=llm-inference"
EXPECTED="a100"

while IFS=$'\t' read -r pod node; do
  actual=$(kubectl get node "$node" -n "$NAMESPACE" -o jsonpath='{.metadata.labels.accelerator.example.com/gpu-family}')
  if [[ "$actual" != "$EXPECTED" ]]; then
    printf 'MISMATCH pod=%s node=%s expected=%s actual=%s
' "$pod" "$node" "$EXPECTED" "$actual"
  fi
done < <(kubectl get pods -n "$NAMESPACE" -l "$SELECTOR" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"
"}{end}')

If drift is found, first confirm replacement capacity, then roll the replacement. Deleting pods directly may interrupt service; a PodDisruptionBudget can constrain voluntary disruptions:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llm-inference
  namespace: NAMESPACE
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: llm-inference

The PDB does not protect against node failures. With a single replica, minAvailable: 1 blocks ordinary evictions, so design it together with real availability goals.

Monitoring and Alerting

Observe Pending and unschedulable pods at the scheduler layer; device‑level availability can be monitored via DCGM Exporter. Use the actual exporter metrics and labels:

sum by (namespace) (
  kube_pod_status_phase{namespace="NAMESPACE",phase="Pending"} == 1
)

max by (namespace, pod) (
  kube_pod_status_unschedulable{namespace="NAMESPACE"}
)

GPU memory usage ratio:

sum by (node) (DCGM_FI_DEV_FB_USED) /
sum by (node) (DCGM_FI_DEV_FB_TOTAL)

Low GPU utilization without requests may be normal; combine with queue length, request latency and Ready status. If the managed cluster does not expose scheduler metrics, fall back to events and kube‑state‑metrics without expanding control‑plane permissions.

Canary and Rollback

Canary one replica, verify device health, endpoint and request path. Changing the PodTemplate triggers a rolling update; with no GPU redundancy, control maxSurge carefully. On failure, inspect history and roll back; the rollback creates a new revision, so ensure the previous ReplicaSet still exists:

kubectl rollout history deployment/llm-inference -n NAMESPACE
kubectl rollout undo deployment/llm-inference -n NAMESPACE --to-revision=REVISION
kubectl rollout status deployment/llm-inference -n NAMESPACE --timeout=10m
kubectl get pods -n NAMESPACE -l app=llm-inference -o wide

If the rollback remains Pending, the old template’s node resources may have changed. Store declarative manifests in version control; temporary exports are for emergency evidence only. Dry‑run the rollback file before applying:

kubectl apply -n NAMESPACE -f llm-inference.before.yaml \
  --dry-run=server -o yaml > /tmp/rollback.rendered.yaml
kubectl diff -n NAMESPACE -f llm-inference.before.yaml || true

Before any release, be able to answer: Are GPU extension resources registered? Who maintains model‑specific labels? Are hard constraints non‑negotiable? Do taints and tolerations match? Is replica distribution enforced? Is there GPU headroom during rollout? How to distinguish Pending, invisible devices, and CUDA OOM? Are rollback configurations and revisions usable?

Closing the loop requires verifying hardware facts, establishing controlled labels and taints, expressing requirements with hard and soft constraints, and then confirming through events, device probes, application health checks and monitoring metrics. Root‑cause analysis must rely on reproducible logs, metrics, events or configuration diffs, not merely on a pod reaching Running.