Advanced Kubernetes Scheduling: Pod Affinity, Anti‑Affinity, and Topology Spread Constraints

Distinguish three “distance” concepts

Scheduling can be described as “spread deployment”, but there are three distinct notions of distance:

Choose nodes with hardware or isolation labels – use nodeSelector or NodeAffinity (e.g., GPU, specific CPU, compliant node pool). Common misuse: replacing node labels with Pod anti‑affinity.

Place a pod near or far from another pod – use PodAffinity (e.g., cache proxy with local data service) or PodAntiAffinity. Common misuse: ignoring resource competition and fault domains.

Avoid co‑location of similar replicas – use PodAntiAffinity or topologySpreadConstraints (e.g., high‑availability API with multiple replicas). Common misuse: setting required rules stricter than the number of available nodes.

Control cross‑domain replica skew – use topologySpreadConstraints (e.g., distribute 6 replicas evenly across 3 zones). Common misuse: assuming it automatically creates new nodes or zones.

Start with scheduling evidence

Before modifying affinity, verify why a Deployment is Pending. Typical causes include insufficient CPU, unsatisfied node affinity, PVC topology limits, image‑pull failures, or disabled preemption. Scheduler events and node labels are primary evidence.

# List nodes with hostname and zone labels
kubectl get nodes -n <namespace> -L kubernetes.io/hostname -L topology.kubernetes.io/zone -L <node-label>
# Show node details and taints
kubectl describe nodes -n <namespace> | rg -n 'Name:|Taints:|Allocated resources:|topology.kubernetes.io/zone'
# List pods with wide output
kubectl get pods -n <namespace> -o wide

Labels are not trustworthy by default; use protected prefixes (e.g., node.kubernetes.io/) for security‑critical attributes. A matching affinity does not guarantee scheduling if the pod does not tolerate a NoSchedule taint.

# Find FailedScheduling events
kubectl get pods -n <namespace> --field-selector=status.phase=Pending -o wide
kubectl describe pod -n <namespace> <pod-name>
kubectl get events -n <namespace> --sort-by=.lastTimestamp | rg 'FailedScheduling|Preempt|Unschedulable'

Do not remove constraints one by one; instead, identify the least‑compatible rule and relax it.

Node labels and NodeAffinity

Use NodeAffinity for stable, auditable node attributes such as CPU architecture, verified GPU model, compliance pools, network devices, or storage locality. Avoid temporary labels for short‑term fixes.

# Dry‑run label change on a node
kubectl get node -n <namespace> <node-name> --show-labels
kubectl describe node -n <namespace> <node-name> | rg -n 'Taints:|Labels:|Unschedulable:'
kubectl label node -n <namespace> <node-name> <node-label>=true --dry-run=server

Example of a simple nodeSelector deployment (hard constraint):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: <namespace>
spec:
  replicas: 3
  selector:
    matchLabels:
      app: <app-label>
  template:
    metadata:
      labels:
        app: <app-label>
    spec:
      nodeSelector:
        <node-label>: "true"
      containers:
      - name: api
        image: <image>@sha256:<digest>
        resources:
          requests:
            cpu: 250m
            memory: 512Mi
          limits:
            cpu: "1"
            memory: 1Gi

Multiple matchExpressions within a single nodeSelectorTerm are ANDed; separate terms are ORed. Mis‑configuring this can unintentionally eliminate all candidates.

# Preferred NodeAffinity (soft constraint)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: <namespace>
spec:
  replicas: 3
  selector:
    matchLabels:
      app: <app-label>
  template:
    metadata:
      labels:
        app: <app-label>
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - <preferred-zone>
      containers:
      - name: api
        image: <image>@sha256:<digest>

Preferred rules are not guarantees; the scheduler still respects hard constraints and may place pods elsewhere due to resource pressure or other plugins.

Pod affinity and anti‑affinity

PodAffinity and PodAntiAffinity use labelSelector to find “reference Pods” and then apply a topologyKey (e.g., topology.kubernetes.io/zone for zone‑level HA or kubernetes.io/hostname for node‑level isolation).

# Required PodAntiAffinity (hard rule)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: <namespace>
spec:
  replicas: 3
  selector:
    matchLabels:
      app: <app-label>
  template:
    metadata:
      labels:
        app: <app-label>
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: <app-label>
            topologyKey: kubernetes.io/hostname
      containers:
      - name: api
        image: <image>@sha256:<digest>

This rule forces each matching pod onto a different hostname. Ensure the number of eligible nodes exceeds the replica count; otherwise pending Pods are expected.

# Preferred PodAntiAffinity (soft rule)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: <namespace>
spec:
  replicas: 3
  selector:
    matchLabels:
      app: <app-label>
  template:
    metadata:
      labels:
        app: <app-label>
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: <app-label>
              topologyKey: topology.kubernetes.io/zone
      containers:
      - name: api
        image: <image>@sha256:<digest>

Preferred anti‑affinity is useful when strict zone distribution would block a rollout; it expresses a preference without making the deployment unschedulable.

# PodAffinity example (prefer same zone as cache)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache-client
  namespace: <namespace>
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cache-client
  template:
    metadata:
      labels:
        app: cache-client
    spec:
      affinity:
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 50
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: shared-cache
              topologyKey: topology.kubernetes.io/zone
      containers:
      - name: client
        image: <image>@sha256:<digest>

Affinity reduces cross‑zone latency but should not be used as a hard availability guarantee.

Topology spread constraints

topologySpreadConstraints

directly controls replica balance. maxSkew defines the allowed difference in pod count per topology domain; whenUnsatisfiable can be DoNotSchedule (hard) or ScheduleAnyway (soft).

# Spread across hostname (hard) and zone (soft)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: <namespace>
spec:
  replicas: 6
  selector:
    matchLabels:
      app: <app-label>
  template:
    metadata:
      labels:
        app: <app-label>
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: <app-label>
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: <app-label>
      containers:
      - name: api
        image: <image>@sha256:<digest>

When multiple hard constraints coexist, the candidate set may become empty (e.g., 6 replicas on 2 nodes in a single zone). Always calculate capacity versus desired replica count before applying constraints.

# Require at least three zones
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: <namespace>
spec:
  replicas: 3
  selector:
    matchLabels:
      app: <app-label>
  template:
    metadata:
      labels:
        app: <app-label>
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        minDomains: 3
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: <app-label>
      containers:
      - name: api
        image: <image>@sha256:<digest>

minDomains

is supported only in newer Kubernetes versions; it enforces a minimum number of distinct topology domains (e.g., three zones). If the cluster cannot satisfy this, the deployment will remain pending.

Coordinate with PDB, PriorityClass, and capacity

PodDisruptionBudgets (PDB) limit voluntary disruptions; they do not protect against node failures. Horizontal Pod Autoscaler (HPA) changes replica counts, which re‑evaluates topology rules. PriorityClass influences preemption but does not replace capacity planning.

# Example PDB for a three‑replica service
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: <namespace>
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: <app-label>

# High‑priority class for critical services
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: online-critical
description: "Only for reviewed critical online services"
value: 100000
globalDefault: false

# Deployment referencing the priority class
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: <namespace>
spec:
  replicas: 3
  selector:
    matchLabels:
      app: <app-label>
  template:
    metadata:
      labels:
        app: <app-label>
    spec:
      priorityClassName: online-critical
      terminationGracePeriodSeconds: 30
      containers:
      - name: api
        image: <image>@sha256:<digest>
        resources:
          requests:
            cpu: 250m
            memory: 512Mi

High priority can cause lower‑priority workloads to be evicted; assign values carefully and only after capacity review.

Observability and metrics

Use Prometheus to monitor unschedulable pods and per‑node pod distribution. Adjust metric names to match the exporter in use.

# Unschedulable pods per namespace
sum by (namespace, pod) (kube_pod_status_unschedulable{namespace="<namespace>"})

# Pod count per node for a given app regex
count by (node) (kube_pod_info{namespace="<namespace>", pod=~"<app-pod-regex>"})

Rollback principles

Affinity, anti‑affinity, and topology spread are part of the pod template; changing them triggers a rolling update. To roll back safely, use a known‑good Git revision or a stored YAML file and apply it with kubectl diff before kubectl apply.

# Show rollout history and undo to a specific revision
kubectl rollout history -n <namespace> deployment/api
kubectl rollout undo -n <namespace> deployment/api --to-revision=<revision>
kubectl rollout status -n <namespace> deployment/api --timeout=180s
kubectl get pods -n <namespace> -l app=<app-label> -o wide

After rollback, verify pod distribution, PDB status, and business endpoints; a successful rollout does not guarantee the desired topology.