Deep Dive into Kubelet: podManager, podWorkers, and workQueue Mechanics

Overview of podManager and podWorkers

Kubelet maintains two central data structures, podManager and podWorkers , which aim to keep pod information consistent. When a pod is force‑deleted, its entry is removed from podManager but the podWorkers still runs the normal deletion flow, leading to a temporary mismatch where podManager lacks the pod while podWorkers still references it.

Newly created pods are first stored in podManager and later added to podWorkers during startup.

MirrorPod handling

Within podManager there are two maps: podByFullName and mirrorPodByFullName. The former stores ordinary Pod objects, the latter stores their corresponding MirrorPod objects.

podByFullName       map[string]*v1.Pod
mirrorPodByFullName map[string]*v1.Pod

Pods can originate from three sources: file , http , and apiserver . Pods reported by the apiserver are called “regular pods”; pods from the other two sources are “static pods”. Kubelet generates a mirror pod in the apiserver for each static pod, creating a one‑to‑one replica that differs only in uid and an extra annotation kubernetes.io/config.mirror.

Deleting a static pod created from a file requires removing the source file, which in turn deletes both the static pod and its mirror pod.

Static pod removal workflow

The podConfig component detects the file removal and triggers a SyncLoop REMOVE.

podWorker runs syncTerminatingPod, stopping containers and the sandbox.

PLEG observes the sandbox and container exit.

podWorker executes syncTerminatedPod, removing the cgroup, updating the mirror pod status, and waiting for volume unmount.

The mirror pod status update is observed.

The housekeeping routine cleans up the mirror pod's volume directory.

The system detects the mirror pod's deletion from the apiserver. garbageCollector removes the sandbox, containers, and pod log directory.

podManager API

podManager stores the collection of pods that should run on the node, including admitted pods and mirror pods. It receives data from the apiserver, static files, and other sources, writing updates to the podUpdateCh channel.

GetPodByFullName(podFullName string) (*v1.Pod, bool)
GetPodByName(namespace, name string) (*v1.Pod, bool)
GetPodByUID(types.UID) (*v1.Pod, bool)
GetPodByMirrorPod(*v1.Pod) (*v1.Pod, bool)
GetMirrorPodByPod(*v1.Pod) (*v1.Pod, bool)
GetPodAndMirrorPod(*v1.Pod) (pod, mirrorPod *v1.Pod, wasMirror bool)
GetPods() []*v1.Pod
GetPodsAndMirrorPods() (allPods []*v1.Pod, allMirrorPods []*v1.Pod, orphanedMirrorPodFullnames []string)
TranslatePodUID(uid types.UID) kubetypes.ResolvedPodUID
GetUIDTranslations() (podToMirror map[kubetypes.ResolvedPodUID]kubetypes.MirrorPodUID, mirrorToPod map[kubetypes.MirrorPodUID]kubetypes.ResolvedPodUID)

AddPod(pod *v1.Pod)
UpdatePod(pod *v1.Pod)
RemovePod(pod *v1.Pod)

The article then walks through how these operations are processed by podWorkers via the UpdatePod method, which selects an UpdatePodOptions variant based on whether the pod is a mirror pod.

podWorkers internals

podWorkers drives the pod lifecycle by running a dedicated goroutine ( podWorkerLoop) for each pod. The interface is defined as:

type PodWorkers interface {
    // Sends a change to the podWorker; the pod is processed FIFO in its own goroutine.
    // The podUID is the key for the goroutine.
    // syncTerminatingPod is called when the pod is marked for termination; subsequent UpdatePod calls are ignored until termination completes.
    UpdatePod(options UpdatePodOptions)
}

The UpdatePodOptions struct contains the update type, the target pod, an optional mirror pod, a possible running pod reference, and optional kill options:

type UpdatePodOptions struct {
    // Update type (create, update, sync, kill).
    UpdateType kubetypes.SyncPodType
    // Pod to update. Required.
    Pod *v1.Pod
    // MirrorPod is the mirror pod if Pod is a static pod. Optional when UpdateType is kill or terminated.
    MirrorPod *v1.Pod
    // RunningPod is the runtime pod that no longer exists; only set during PodCleanup.
    RunningPod *kubecontainer.Pod
    // KillPodOptions overrides the default termination behavior or updates pod status after completion.
    KillPodOptions *KillPodOptions
}

The concrete podWorkers struct holds a map of per‑UID update channels, a workQueue, and a podSyncer:

type podWorkers struct {
    podUpdates map[types.UID]chan struct{}
    workQueue  queue.WorkQueue
    podSyncer  podSyncer
}

workQueue implementation

The WorkQueue interface provides two core methods: GetWork(), which dequeues all ready items, and Enqueue(item, delay), which inserts or overwrites an item with an optional delay.

type WorkQueue interface {
    // GetWork dequeues and returns all ready items.
    GetWork() []types.UID
    // Enqueue inserts a new item or overwrites an existing item.
    Enqueue(item types.UID, delay time.Duration)
}

The default implementation, basicWorkQueue, stores items in a map keyed by UID with a timestamp value:

type basicWorkQueue struct {
    clock clock.Clock
    lock  sync.Mutex
    queue map[types.UID]time.Time
}

syncPod and overall Kubelet flow

The syncPod() function is the heart of Kubelet, handling both pod creation and deletion. It coordinates with other Kubelet components such as volumeManager and podManager to keep the pod state consistent between the node and etcd. The article notes that because syncPod() touches many subsystems, a detailed analysis of those components is deferred.

A lingering question is whether parallel processing of pods by multiple goroutines could cause resource‑allocation contention, which the author leaves for future investigation.