How Kubelet Manages Pod Lifecycle: Deep Dive into Sync Loop and Workers

1. Getting Pod Configuration

Kubelet obtains the pod specifications for the node from three primary sources: the API server, a directory of static configuration files, and a dedicated HTTP endpoint. It polls these sources periodically, detects changes, and updates its internal PodConfig object.

type PodConfig struct {
    pods    *podStorage
    mux     *config.Mux
    // channel of denormalized changes passed to listeners
    updates chan kubetypes.PodUpdate
    ...
}

The mux component listens to the three sources ( apiserver, file, http) and merges their latest states. When a difference is detected, a PodUpdate is created and sent on the updates channel.

type PodUpdate struct {
    Pods   []*v1.Pod
    Op     PodOperation
    Source string
}

The Op field indicates the type of change, such as ADD, UPDATE or REMOVE, allowing downstream components to react accordingly.

2. Sync Loop

After initialization Kubelet runs its main loop syncLoop, which continuously watches the updates channel and several internal channels. Each event is delegated to a specific handler.

func (kl *Kubelet) syncLoop(updates <-chan kubetypes.PodUpdate, handler SyncHandler) {
    for {
        if !kl.syncLoopIteration(...) {
            break
        }
    }
}

func (kl *Kubelet) syncLoopIteration(configCh <-chan kubetypes.PodUpdate, ...) bool {
    select {
    case u, open := <-configCh:
        switch u.Op {
        case kubetypes.ADD:
            handler.HandlePodAdditions(u.Pods)
        case kubetypes.UPDATE:
            handler.HandlePodUpdates(u.Pods)
        }
    case e := <-plegCh:
        handler.HandlePodSyncs([]*v1.Pod{pod})
    case <-syncCh:
        podsToSync := kl.getPodsToSync()
        if len(podsToSync) == 0 { break }
        handler.HandlePodSyncs(podsToSync)
    case update := <-kl.livenessManager.Updates():
        if update.Result == proberesults.Failure {
            handler.HandlePodSyncs([]*v1.Pod{pod})
        }
    case <-housekeepingCh:
        handler.HandlePodCleanups()
    }
    return true
}

The loop processes five main event types:

Configuration changes from configCh (additions, updates, removals).

Container lifecycle events from plegCh (e.g., ContainerStarted).

Periodic sync ticks from syncCh (default every second).

Liveness probe failures from livenessManager.

House‑keeping clean‑ups from housekeepingCh (default every two seconds).

3. Pod Workers

For each pod Kubelet creates a dedicated podWorker goroutine. The worker owns a single‑element channel of type UpdatePodOptions and registers this channel in the podUpdates map of the podWorkers structure, enabling the main loop to forward updates directly to the responsible worker.

type podWorkers struct {
    podUpdates               map[types.UID]chan UpdatePodOptions
    isWorking                map[types.UID]bool
    lastUndeliveredWorkUpdate map[types.UID]UpdatePodOptions
    workQueue                queue.WorkQueue
    syncPodFn                syncPodFnType
    podCache                 kubecontainer.Cache
    ...
}

If a worker is already processing an update when a new one arrives, the newest update is stored in lastUndeliveredWorkUpdate and applied immediately after the current update finishes. After each update the pod is re‑queued in workQueue with a delay; once the delay expires the worker processes the pod again, forming a closed‑loop synchronization cycle.

4. Container Runtime SyncPod

The final step is performed by the container‑runtime abstraction. Kubelet calls SyncPod on the runtime manager, which first computes the required actions by comparing the current podStatus with the desired pod spec.

func (m *kubeGenericRuntimeManager) SyncPod(pod *v1.Pod, _ v1.PodStatus, podStatus *kubecontainer.PodStatus, pullSecrets []v1.Secret, backOff *flowcontrol.Backoff) (result kubecontainer.PodSyncResult) {
    actions := computePodActions(pod, podStatus)
    // actions is a podActions struct
    ...
}

The computed podActions struct enumerates concrete operations such as killing the sandbox, creating a new sandbox, killing containers, starting init containers, and launching regular containers.

type podActions struct {
    KillPod            bool
    CreateSandbox      bool
    SandboxID          string
    Attempt            uint32
    ContainersToKill   map[kubecontainer.ContainerID]containerToKillInfo
    NextInitContainerToStart *v1.Container
    ContainersToStart []int
}

KillPod

and CreateSandbox indicate whether the existing sandbox must be removed and a new one created. SandboxID identifies the sandbox; an empty value means this is the first creation. Attempt counts how many times the sandbox has been recreated. ContainersToKill lists containers that must be stopped (e.g., due to configuration change or health‑check failure). NextInitContainerToStart points to the next init container that needs to be created and started. ContainersToStart holds the indexes of regular containers that are not yet running and should be started.

SyncPod then sequentially invokes the corresponding runtime interfaces ( KillSandbox, CreateSandbox, StartContainer, StopContainer, etc.) to bring the actual container state in line with the desired state.

5. Overall Flow

Kubelet continuously fetches pod specifications, converts configuration changes into PodUpdate events, routes those events through syncLoop to the appropriate pod workers, and finally delegates concrete container actions to the underlying container runtime via SyncPod. The cycle repeats on every configuration change or periodic sync tick, ensuring that the node’s pods converge toward their intended state.

References

Kubernetes source code (v1.9.4, commit bee2d1505c4fe820744d26d41ecd3fdd4a3d6546) – https://github.com/YaoZengzeng/kubernetes

What even is a kubelet? – http://kamalmarhubi.com/blog/2015/08/27/what-even-is-a-kubelet/