Master Kubernetes Scheduling: 15 Real-World Scenarios & Configurations

Kubernetes scheduling is a critical component that ensures Pods run on appropriate nodes, improving resource utilization, load balancing, and high availability. This article dives into practical scheduling scenarios with configuration examples and best practices.

1. Basic Scenario – NodeSelector

Scenario description: We have nodes labeled with SSD disks and want to schedule Pods requiring high‑performance storage onto these nodes.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: high-performance-pod
spec:
  containers:
  - name: my-container
    image: my-image
  nodeSelector:
    disktype: ssd

2. Advanced Scenario – Node Affinity

Scenario description: We want to schedule tasks that need a GPU onto nodes labeled with a GPU.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: my-container
    image: my-image
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu
            operator: In
            values:
            - "true"

3. Resource Allocation – Pod Priority and PriorityClass

Scenario description: To ensure critical workloads have higher priority, define a PriorityClass and apply it to Pods.

PriorityClass configuration:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "High priority class"

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: high-priority-pod
spec:
  containers:
  - name: my-container
    image: my-image
  priorityClassName: high-priority

4. Prevent Pods from Running on the Same Node – Pod Anti‑Affinity

Scenario description: Using pod anti‑affinity ensures Pods in the same group are not scheduled onto the same node, improving high availability.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: anti-affinity-pod
spec:
  containers:
  - name: my-container
    image: my-image
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - web
        topologyKey: kubernetes.io/hostname

5. Multi‑Replica Topology Distribution – Pod Topology Spread

Scenario description: Ensure multiple Pods of the same application are spread across different topology domains to increase availability.

Deployment configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: "app"
                operator: In
                values:
                - "web"
            topologyKey: "kubernetes.io/hostname"

6. Node Taints and Pod Tolerations

Scenario description: By tainting nodes, only Pods with matching tolerations can be scheduled onto those nodes.

Node configuration:

apiVersion: v1
kind: Node
metadata:
  name: node1
spec:
  taints:
  - key: special
    value: unique
    effect: NoSchedule

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: toleration-pod
spec:
  containers:
  - name: my-container
    image: my-image
  tolerations:
  - key: "special"
    operator: "Equal"
    value: "unique"
    effect: "NoSchedule"

7. Custom Scheduler – Custom Scheduler Example

Scenario description: Build a custom scheduler to meet specific scheduling needs, such as business rules or special hardware requirements.

Custom scheduler code (Go):

// my_scheduler.go
package main

import (
    "k8s.io/kubernetes/pkg/scheduler"
    "k8s.io/kubernetes/pkg/scheduler/framework"
    "k8s.io/kubernetes/pkg/scheduler/framework/plugins/defaultbinder"
    "k8s.io/kubernetes/pkg/scheduler/framework/plugins/defaultpreemption"
    "k8s.io/kubernetes/pkg/scheduler/framework/plugins/names"
)

const (
    // YourSchedulerName is the name of your custom scheduler
    YourSchedulerName = "my-scheduler"
)

// New initializes a new scheduler with your custom plugins
func New() *scheduler.Config {
    return &scheduler.Config{
        Client:              scheduler.NewHTTPClient(),
        SchedulerName:       YourSchedulerName,
        PercentageOfNodesToScore: 0.25,
        Profiles: []scheduler.Profile{{
            Name: YourSchedulerName,
            Plugins: []scheduler.Plugin{
                defaultpreemption.Name: defaultpreemption.New,
                defaultbinder.Name:     defaultbinder.New,
                names.NewNodeResourcesFit(),
                names.NewNodePorts(),
                names.NewNodeAffinity(YourSchedulerName),
                names.NewNodeAffinityPriority(YourSchedulerName),
            },
        }},
    }
}

func main() {
    // Use the New() function to create a new scheduler with your custom plugins
    config := New()
    command := app.NewSchedulerCommand(
        app.WithConfig(config),
    )
    f := command.Flags()
    f.AddGoFlagSet(flag.CommandLine)

    if err := command.Execute(); err != nil {
        os.Exit(1)
    }
}

Compile and run the custom scheduler:

go build my_scheduler.go
./my_scheduler

8. Pod Priority and Preemption

Scenario description: Setting pod priority and preemption policies ensures critical tasks are handled first.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: priority-pod
spec:
  containers:
  - name: my-container
    image: my-image
  priorityClassName: high-priority

9. Resource Limits and Requests

Scenario description: Defining resource limits and requests allows the scheduler to better optimize resource usage.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: resource-pod
spec:
  containers:
  - name: my-container
    image: my-image
    resources:
      limits:
        cpu: "2"
        memory: "1Gi"
      requests:
        cpu: "1"
        memory: "500Mi"

10. Affinity and Anti‑Affinity Rules

Scenario description: Use affinity and anti‑affinity rules to ensure Pods are placed on specific nodes or avoid being scheduled together.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: affinity-pod
spec:
  containers:
  - name: my-container
    image: my-image
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: security
            operator: In
            values:
            - "high"
        topologyKey: kubernetes.io/hostname

11. Pod Disruption Budget

Scenario description: Use a PodDisruptionBudget to limit the number of Pods that can be disrupted during maintenance, ensuring system stability.

PodDisruptionBudget configuration:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web

12. Horizontal Pod Autoscaler

Scenario description: The HorizontalPodAutoscaler automatically adjusts the number of Pods based on CPU usage or other metrics.

HPA configuration:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

13. Pod Overhead

Scenario description: Setting pod overhead informs the scheduler of additional resources a Pod needs, preventing over‑scheduling on a node.

Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: overhead-pod
spec:
  containers:
  - name: my-container
    image: my-image
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
    overhead:
      podFixed: 100Mi
      ephemeral-storage: 1Gi

14. Node‑Local DNS Cache

Scenario description: Enabling a local DNS cache on nodes improves DNS query performance.

Kubelet configuration:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDomain: cluster.local
featureGates:
  CoreDNSLocalCache: true

15. Pod Priority Classes

Scenario description: Using Pod priority classes categorizes Pods by priority, ensuring critical workloads are scheduled first.

PriorityClass configuration:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "High priority class"
preemptionPolicy: PreemptLowerPriority

Conclusion

These scenarios cover a range from basic to advanced Kubernetes scheduling use cases. Choose the appropriate scenario to configure your cluster for optimal resource utilization and performance, and adjust settings to match your specific business and performance requirements.