Mastering Kubernetes Control Plane: etcd, API Server, Scheduler & More

etcd is a fast, distributed, consistent key‑value store that serves as the persistent storage for Kubernetes objects such as pods, replication controllers, secrets, and services; it is the sole source of cluster state and metadata, accessed directly only by the API Server.

etcd provides a watch mechanism that notifies observers of key changes, enabling components like the API Server to react to state updates.

Should the number of etcd instances be odd? In high‑availability setups, 3, 5, or 7 instances are typical because etcd uses the Raft consensus algorithm, which requires a majority to progress; with an even number, a failure could prevent the cluster from achieving a quorum.

API Server

The API Server is the only component that talks directly to etcd; all other Kubernetes components and clients must interact with the cluster state through it.

Provides a consistent way to store objects in etcd.

Validates objects to prevent malformed configurations from being written directly to etcd.

Exposes a RESTful API for creating, updating, and deleting resources.

Implements optimistic concurrency control to avoid overwriting concurrent updates.

Performs authentication and authorization for incoming requests.

Executes admission control (e.g., AlwaysPullImages, DefaultStorageClass, ResourceQuota) when creating or modifying resources.

Implements a watch mechanism for clients to monitor resource changes.

Controller Manager

Controllers monitor the cluster state and continuously work to move the current state toward the desired state, each tracking at least one Kubernetes resource type.

Examples of controllers include ReplicationController, ReplicaSet, DaemonSet, Job, Deployment, StatefulSet, Node, Service, Endpoints, Namespace, and PersistentVolume controllers.

Controllers use watches to receive change notifications from the API Server and act on creations, updates, or deletions; they also periodically relist resources to avoid missing events.

The Controller Manager also handles lifecycle functions such as namespace creation, event garbage collection, pod termination, cascading deletion garbage collection, and node garbage collection.

Scheduler

The Scheduler assigns pods to nodes by watching newly created pods without a node assignment, filtering feasible nodes, scoring them, and binding the pod to the highest‑scoring node.

Filter the list of all nodes to obtain a set of nodes that satisfy the pod's scheduling requirements.

Score the filtered nodes, rank them, and select the best one; ties are broken using a round‑robin approach.

Scheduling decisions consider factors such as resource requests, node labels matching pod node selectors, port conflicts, pod tolerations for node taints, and node affinity or anti‑affinity rules.

Worker Node Components

Kubelet

Kubelet runs on each cluster node as an agent that ensures containers defined in pods are running.

Registers the node with the API Server by creating a Node resource.

Continuously watches pods scheduled to the node.

Starts pod containers using the configured container runtime.

Reports container status, events, and resource usage back to the API Server.

Performs liveness probes, restarts failed containers, and terminates containers when their pod is deleted.

kube-proxy

kube-proxy runs on each node to enable communication between pods, services, and nodes, maintaining up‑to‑date network rules based on Service and Endpoint changes.

Allocates a virtual IP when a Service is created.

API Server notifies kube‑proxy of new Services.

Sets iptables/ipvs rules so that traffic to the Service IP is redirected to one of the backing pods.

Watches the API Server for changes to Services or Endpoints.

Container Runtime

The container runtime is responsible for pulling images, unpacking them into a layered filesystem, preparing mount points, setting metadata (CMD, ENTRYPOINT, seccomp), applying isolation (process, network, filesystem), enforcing resource limits, and invoking syscalls to start containers with correct SELinux/AppArmor settings.

Pulls the required image from a registry if not present locally.

Extracts the image into a copy‑on‑write filesystem.

Prepares the container's mount points.

Sets container metadata such as command overrides and security policies.

Applies isolation mechanisms (cgroups, namespaces).

Enforces resource limits like CPU and memory.

Executes the container's process via a syscall.

Ensures SELinux/AppArmor profiles are correctly applied.

References

RAFT Consensus Algorithm: https://www.geeksforgeeks.org/raft-consensus-algorithm/

Admission Controllers: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

Cascading Deletion Garbage Collection: https://kubernetes.io/docs/concepts/architecture/garbage-collection/