K8s Scheduler Black Box: How TopologySpreadConstraints’ Math Can Trip Engineers

This guide is aimed at engineers who have already used NodeAffinity, Taints & Tolerations, or PriorityClass and still feel that Kubernetes scheduling is missing something.

Conclusion: Anti‑affinity often isn’t enough

Pod anti‑affinity makes a binary decision – can the pod be placed or not – which is a hard constraint. TopologySpreadConstraints, by contrast, decide where a pod should be placed to keep the distribution balanced, a finer‑grained constraint.

Typical scenario: two worker nodes need to run three replicas. With hard anti‑affinity the third pod will likely stay Pending; with soft anti‑affinity the pods may be placed randomly, leading to an uneven spread. TopologySpreadConstraints aim to satisfy both schedulability and even distribution.

The key metric is maxSkew : the allowed difference in pod counts between any two topology domains.

What the scheduler actually computes

topologyKey

– defines the boundary (e.g., kubernetes.io/hostname). labelSelector – selects which pods are counted. maxSkew – the maximum tolerated imbalance.

The most common trap is the “empty topology domain”. The scheduler does not only look at nodes that already have pods; it includes every valid domain that matches the topology key. If any domain is empty, the skew calculation can fail, causing pods to remain Pending even though resources appear sufficient.

In a Kind cluster with one control‑plane node and two workers, the control‑plane node is treated as an empty domain in the calculation, leading to the unexpected Pending state.

Make the scheduler treat only truly usable worker nodes as effective topology domains.

By excluding the empty control‑plane node from the skew calculation, the Pending issue disappears, demonstrating a deeper understanding of the scheduler’s statistical scope.

Going beyond YAML: custom scheduler plugins

When business requirements need priority nodes, forbidden nodes, or weighted labels, native YAML constraints are insufficient. The Kubernetes Scheduling Framework allows developers to extend the scheduler with plugins.

Two typical plugins are:

Filter plugin – decides whether a node is eligible.

Score plugin – ranks eligible nodes to pick the best one.

Combined, they expose most of the scheduler’s “black box”. The plugin pipeline is: PreFilter → Filter → Score → Bind Each stage is clearly defined, but the documentation can be intimidating; hands‑on experience reveals the real pitfalls.

Practical challenges when writing Go plugins

The biggest obstacle is not the plugin logic itself but the dependency and version ecosystem. Mismatched version numbers, a failing go mod tidy, changed PluginFactory signatures, or insufficient RBAC permissions can all break the build.

Since Kubernetes v1.XX, the plugin factory function signature changed, rendering many older tutorials unusable.

The true measure of success is getting the whole chain working: compiling the plugin, building the image, loading it into a Kind cluster, and seeing the scheduling effect.

Why this article matters

It goes beyond conceptual explanations and shows exactly how topology constraints are calculated, why empty domains cause Pending pods, how to register scheduler plugins, how to avoid Go dependency traps, and how to quickly validate everything in a local Kind environment.