Mastering Kubernetes Scheduler: From Extenders to the New Scheduling Framework

Kubernetes has become the de‑facto standard for container cluster management, providing automated deployment, operation, and resource scheduling. As workloads expand beyond web services to include machine learning, high‑performance computing, and specialized accelerators, the built‑in Kube‑scheduler faces new challenges in handling heterogeneous resources.

Early solutions

The community initially offered two ways to extend scheduling:

Scheduler Extender : an external HTTP service invoked during Filter, Preempt, Prioritize, and Bind phases. While flexible for small clusters, it suffers from network latency, marshaling overhead, and limited extension points.

Multiple schedulers : developers run a custom scheduler alongside the default one. This can cause resource conflicts, lower cluster utilization, and increase maintenance effort.

Both approaches expose a trade‑off between performance and extensibility, prompting the development of a more robust solution.

Kubernetes Scheduling Framework (v2)

Starting with Kubernetes 1.16, the Scheduling Framework introduces rich extension points that plugins can implement, enabling fine‑grained control without sacrificing performance. The framework splits the scheduling process into two cycles:

Scheduling cycle (synchronous, thread‑safe): includes QueueSort, PreFilter, Filter, PostFilter, PreScore, Scoring, Reserve, Permit.

Binding cycle (asynchronous, thread‑unsafe): includes Bind, PreBind, PostBind, Unreserve.

Each extension point is invoked at a specific stage, allowing plugins to influence decisions such as node ordering, filtering criteria, scoring algorithms, and binding behavior.

Implementing a custom QoS plugin

The following example shows how to create a QueueSort plugin that orders Pods by their Quality‑of‑Service class when priorities are equal.

// QoSSort is a plugin that implements QoS‑based sorting.
type Sort struct{}

// New initializes a new plugin.
func New(_ *runtime.Unknown, _ framework.FrameworkHandle) (framework.Plugin, error) {
    return &Sort{}, nil
}

// Less defines the ordering logic.
func (s *Sort) Less(pInfo1, pInfo2 *framework.PodInfo) bool {
    p1 := pod.GetPodPriority(pInfo1.Pod)
    p2 := pod.GetPodPriority(pInfo2.Pod)
    return (p1 > p2) || (p1 == p2 && compQOS(pInfo1.Pod, pInfo2.Pod))
}

func compQOS(p1, p2 *v1.Pod) bool {
    p1QOS, p2QOS := v1qos.GetPodQOS(p1), v1qos.GetPodQOS(p2)
    if p1QOS == v1.PodQOSGuaranteed {
        return true
    } else if p1QOS == v1.PodQOSBurstable {
        return p2QOS != v1.PodQOSGuaranteed
    }
    return p2QOS == v1.PodQOSBestEffort
}

Register the plugin in cmd/main.go:

func main() {
    rand.Seed(time.Now().UnixNano())
    command := app.NewSchedulerCommand(
        app.WithPlugin(qos.Name, qos.New),
    )
    if err := command.Execute(); err != nil {
        os.Exit(1)
    }
}

Build and run the scheduler:

$ make
$ bin/kube-scheduler --kubeconfig=scheduler.conf --config=./manifests/qos/scheduler-config.yaml

After these steps, the custom QoS plugin participates in the scheduling cycle, influencing pod ordering based on QoS class.

Future work

The Scheduling Framework opens the door for integrating advanced data‑center scheduling concepts such as co‑scheduling, capacity scheduling, dominant‑resource fairness, and multi‑queue management, enabling Kubernetes to handle AI, big‑data, and high‑spec compute workloads more effectively.