How Kubernetes Controllers Automate Cluster Management: A Deep Dive

1 Overview

Kubernetes, as a mature container orchestration system, is offered as a standard service by major cloud providers (AWS, Google Cloud, Alibaba Cloud) and provides features such as gray releases, blue‑green deployments, elastic scaling, automatic restarts, high availability, and comprehensive application, network, and storage management, greatly reducing operational costs.

Kubernetes can manage up to 5,000 nodes and 150,000 Pods in a single cluster thanks to its declarative API programming and controller pattern collaboration. Controllers ensure the current state matches the desired state, acting as the cluster's "central brain" (e.g., ReplicaSet maintains Pod counts, Node controller monitors node health).

2 Controller Pattern Principle

In robotics and automation, a control loop consists of a controller, the controlled system, and sensors. The external system sets a desired state (spec), sensors report the current state (status), and the controller adjusts the system to reduce the difference, forming a closed‑loop negative feedback system.

3 Controller Pattern in Kubernetes

3.1 Kubernetes Controller Components

Kubernetes adopts the controller pattern with built‑in controllers such as Deployment, Service, and StatefulSet, running inside the

kube-controller-manager

. Each controller watches a specific resource type, which has a spec (desired state) and status (current state), and reconciles them.

<code>type Object struct {
    metav1.TypeMeta
    metav1.ObjectMeta
    Spec   ObjectSpec
    Status ObjectStatus
}</code>

All resources contain metadata, spec, and status. Controllers listen to create/update/delete events via the API server and continuously reconcile resources.

<code>for {
    desired := getDesiredState()
    current := getCurrentState()
    makeChanges(desired, current)
}</code>

The controller logic is built from three core components: Reflector, Informer, and Indexer.

Reflector uses List & Watch to fetch resource data, storing updates in a delta queue to avoid duplicate records.

Informer processes delta records, triggers registered event handlers, and updates a local cache. It uses an Indexer to store objects with keys (typically namespace/name) for fast lookup.

Workers pull items from the work queue, fetch the latest object state, and perform create/update actions or call external services. Failed items are re‑queued for retry.

3.2 Example – ReplicaSet Scaling

ReplicaSet maintains a set of Pod replicas. When the desired replica count is increased (e.g., from 2 to 3), the ReplicaSet controller watches for changes, enqueues the key

default/App

, and a worker creates a new Pod owned by the ReplicaSet.

Subsequent Pod add events are added to the delta queue, cached by the Indexer, and the controller updates the ReplicaSet status so that

spec.replicas

matches

status.replicas

.

4 Reflections on Programming with the Controller Pattern

Kubernetes achieves container orchestration through the controller pattern, monitoring ETCD‑stored API resources and reacting to changes. This declarative approach contrasts with imperative programming, which requires explicit step‑by‑step commands.

4.1 Comparison of Two Programming Styles

Imperative programming is like a teacher giving direct commands to students; it focuses on the process and requires developers to specify every action. Declarative programming resembles a manager setting goals; the system determines how to reach the desired state, emphasizing outcomes over procedures.

Imperative systems often suffer from complex error handling, inconsistency between inspection and operational logic, and concurrency issues that require locking, reducing efficiency.

Declarative systems record the current and desired states, are idempotent, and allow concurrent operations without explicit locks, leading to higher stability and scalability.

4.2 Declarative Practice

Prometheus, an open‑source monitoring system, uses YAML configuration files. To simplify rule creation, a visual Prometheus configuration system can be built using a declarative approach.

alert: name of the alert rule.

expr: PromQL expression that triggers the alert.

for: optional evaluation time window.

labels: custom labels attached to the alert.

annotations: additional information displayed with the alert.

Alert rules can be stored in relational tables (e.g., MySQL) as

rule

and

expr

, mirroring Kubernetes' declarative resources. A controller periodically checks the desired rule file against the active Prometheus configuration and updates it atomically, ensuring idempotent, fault‑tolerant operation.

5 Conclusion

Kubernetes is a complex distributed system designed around declarative API resources managed by the

kube-controller-manager

. Controllers drive the cluster toward the desired state, enabling automation and unattended operation. The process can be abstracted as:

Observe: monitor resource events to obtain the current state.

Analyze: detect differences between current and desired states.

Execute: perform actions that move the resource toward the target state.

Because controllers and resources are extensible, custom controllers (Operators) can manage stateful applications like MySQL, Kafka, or ETCD, providing full lifecycle automation, scaling, backup, and monitoring. The declarative controller model can also inspire the design of stable, precise, and fault‑tolerant systems beyond Kubernetes.