Unveiling Kubelet: How Kubernetes Brings Pods to Life with Go Concurrency

Why Kubelet Is the Core of Kubernetes

Kubelet is the most complex core component of Kubernetes. It is not merely a tool that runs containers; it is an OS‑level intelligent agent on each node built with Go’s concurrency primitives, handling container creation, network setup, storage mounting, health probing, and state synchronization with the API server.

Key responsibilities include:

Receiving Pod specifications

Creating containers via containerd or cri‑o

Configuring CNI networking

Mounting CSI storage

Running health probes

Managing container lifecycles

Reporting status to the API server

The internal structure can be seen in the source:

type Kubelet struct {
    // configuration
    kubeletConfiguration kubeletconfiginternal.KubeletConfiguration
    podManager          kubepod.Manager // manages desired Pod state
    podWorkers          PodWorkers       // ensures per‑Pod serial scheduling
    probeManager        prober.Manager   // health probing system
    volumeManager       volumemanager.VolumeManager
    containerGC         kubecontainer.GC
    imageManager        images.ImageGCManager
    pleg                pleg.PodLifecycleEventGenerator
    eventedPleg         pleg.PodLifecycleEventGenerator
    podCache            kubecontainer.Cache
    containerRuntime    kubecontainer.Runtime // CRI implementation
}

Pod Lifecycle on a Node

A single diagram illustrates the full life of a Pod on a node:

API Server
    │ watch
    ▼
Kubelet
  ├─ PodManager (stores desired Pod state)
  ├─ syncLoop (event‑driven main loop)
  ├─ PLEG (real‑time container state detection)
  ├─ PodWorkers (one queue per Pod)
  ├─ syncPod (core of create/delete containers)
  │   ├─ create sandbox (pause)
  │   ├─ configure CNI network
  │   ├─ create business containers
  │   └─ start probes
  └─ Container Runtime (containerd/cri‑o)

Go Concurrency in Kubelet

The Go concurrency model appears in three main places:

Channel‑driven event system: syncLoop uses Go channels to efficiently dispatch events.

Goroutine isolation: each Pod gets its own worker goroutine, guaranteeing operation isolation.

Mutex protection: shared state is accessed safely via mutexes.

Kubelet Startup Flow (Essential for Troubleshooting)

The simplified startup code shows how core components are initialized and injected into the Kubelet struct:

func createAndInitKubelet(...) (k kubelet.Kubelet, err error) {
    // initialize container runtime (CRI)
    containerRuntime, err := kuberuntime.NewKubeGenericRuntimeManager(...)
    // create PLEG
    pleg := pleg.NewGenericPLEG(containerRuntime, plegChannel)
    // create PodWorkers (one worker per Pod)
    podWorkers := newPodWorkers()
    // create secret and configMap managers
    secretManager := secret.NewManager(...)
    configMapManager := configmap.NewManager(...)
    // create VolumeManager
    volumeManager := volumemanager.NewVolumeManager(...)
    // create container GC
    containerGC := kubecontainer.NewContainerGC(...)
    // create image GC manager
    imageManager := images.NewImageGCManager(...)
    // inject all components
    k := &Kubelet{
        podWorkers:       podWorkers,
        volumeManager:    volumeManager,
        containerGC:      containerGC,
        imageManager:     imageManager,
        pleg:             pleg,
        secretManager:   secretManager,
        configMapManager: configMapManager,
        containerRuntime: containerRuntime,
    }
    return k, nil
}

Key initialization steps include connecting to containerd/CRI, initializing CNI and CSI plugins, starting PodManager, PLEG, syncLoop, and the probe system.

syncLoop – The Event Dispatch Brain

syncLoop is the "cerebral cortex" of Kubelet. Its sole duty is to distribute tasks to PodWorkers for serial execution. It listens to three event types:

Pod updates from the API server

Container state changes from PLEG

Probe results from the probe manager

The core loop uses Go’s select to handle these events efficiently:

func (kl *Kubelet) syncLoop(...) {
    for {
        select {
        case update := <-configCh:
            // Pod configuration changed – dispatch to workers
            kl.dispatchWork(update)
        case plegEvent := <-plegCh:
            // Real container state change (crash, exit, start)
            kl.handlePLEGEvent(plegEvent)
        case runtimeState := <-runtimeStateCh:
            // Container runtime ready
            kl.runtimeReady()
        case <-syncTicker:
            // Periodic forced sync of all Pods
            kl.syncAllPods()
        }
    }
}

This non‑blocking, channel‑based design provides high‑throughput event distribution, efficient communication, and built‑in recovery mechanisms.

PodWorkers – Per‑Pod Serial Execution

Each Pod receives a dedicated worker goroutine and a dedicated queue, ensuring that all operations on that Pod are executed serially, preventing concurrent modifications.

for each podUID {
    worker := go func() {
        for key := range queue {
            syncPod(key)
        }
    }()
}

This guarantees that container creation, deletion, probe setup, network updates, and sandbox recreation never run in parallel for the same Pod, contributing to Kubernetes stability.

syncPod – The Full Container Creation Process

The core of turning a Pod into real containers is implemented in SyncPod:

func (m *kubeGenericRuntimeManager) SyncPod(...) error {
    // Ensure or create sandbox (pause container)
    ensureSandbox()
    // Pull required images
    pullImages()
    // Create containers
    createContainers()
    // Start containers
    startContainers()
    // Initialize probes
    startProbes()
}

This sequence is where a Pod actually becomes a set of running containers.

PLEG – Container Health Radar

PLEG periodically pulls the real state of containers from containerd, detects differences, generates events, and triggers syncLoop to recover unhealthy containers.

Typical issues detected by PLEG include container exit, sandbox crashes, deadlocks, or missing containers. When a problem occurs, the log line "PLEG is not healthy" appears.

// Start spawns a goroutine to relist periodically.
func (g *GenericPLEG) Start() {
    g.runningMu.Lock()
    defer g.runningMu.Unlock()
    if !g.isRunning {
        g.isRunning = true
        g.stopCh = make(chan struct{})
        go wait.Until(g.Relist, g.relistDuration.RelistPeriod, g.stopCh)
    }
}

Engineering Tricks That Make Kubelet Super Stable

Goroutine isolation per component

panic recovery in every loop

All operations are idempotent

CRI calls automatically retry

PodWorkers enforce per‑Pod serial execution

These design choices allow Kubelet to run for years without a single Pod crash taking down the whole node.

Common Production‑Level Debugging Methods

Below are practical commands to diagnose Kubelet issues:

View Kubelet logs: journalctl -u kubelet -f Check container runtime state: crictl ps and crictl logs <id> Detect PLEG problems: look for the log line PLEG is not healthy Debug CNI: inspect /etc/cni/net.d/, /var/run/cni/, and /var/log/cni/ Check volume mount failures: examine /var/lib/kubelet/pods/<uid>/volumes/ Focusing on these logs and directories helps pinpoint the root cause of node‑level failures.

Conclusion

Understanding Kubelet—its syncLoop event system, per‑Pod serial PodWorkers, syncPod creation flow, PLEG health detection, probe framework, and integration with CRI/CNI/CSI—is essential to mastering Kubernetes. The source code reveals that Kubernetes is not magic but a product of meticulous engineering, leveraging Go’s concurrency strengths.