Deep Dive into Kubernetes Pod Creation: Exploring the kubelet Source Code

The author, Hua Zai, shares his experience of studying Kubernetes internals after four years of using the platform, focusing on the source code that handles pod creation in version 1.21.3.

Pod Creation Flow in kubelet

1. The CRUD operations for pods are triggered in pkg/kubelet/kubelet.go within the syncLoop() function.

// syncLoop is the main loop for processing changes. It watches for changes from three channels (file, apiserver, and http) and creates a union of them. For any new change seen, it will run a sync against desired state and running state. If no changes are seen, it synchronizes the last known desired state every sync-frequency seconds. Never returns.
func (kl *Kubelet) syncLoop(updates <-chan kubetypes.PodUpdate, handler SyncHandler) {
    klog.InfoS("Starting kubelet main sync loop")
    // ...
}

Within syncLoop(), the function kl.syncLoopIteration() performs the specific pod operations.

kl.syncLoopMonitor.Store(kl.clock.Now())
if !kl.syncLoopIteration(updates, handler, syncTicker.C, housekeepingTicker.C, plegCh) {
    break
}

2. syncLoopIteration receives several important channels:

// Arguments:
// 1. configCh:   a channel to read config events from
// 2. handler:    the SyncHandler to dispatch pods to
// 3. syncCh:     a channel to read periodic sync events from
// 4. housekeepingCh: a channel to read housekeeping events from
// 5. plegCh:    a channel to read PLEG updates from

func (kl *Kubelet) syncLoopIteration(configCh <-chan kubetypes.PodUpdate, handler SyncHandler,
    syncCh <-chan time.Time, housekeepingCh <-chan time.Time, plegCh <-chan *pleg.PodLifecycleEvent) bool {
    select {
    case u, open := <-configCh:
        if !open {
            klog.ErrorS(nil, "Update channel is closed, exiting the sync loop")
            return false
        }
        // ...
    }
    // ...
}

3. SyncHandler is an interface implemented by kubelet for testability, defining methods for common pod operations:

type SyncHandler interface {
    HandlePodAdditions(pods []*v1.Pod)
    HandlePodUpdates(pods []*v1.Pod)
    HandlePodRemoves(pods []*v1.Pod)
    HandlePodReconcile(pods []*v1.Pod)
    HandlePodSyncs(pods []*v1.Pod)
    HandlePodCleanups() error
}

4. Pod operations are represented by constants such as ADD, DELETE, UPDATE, etc., each invoking the corresponding handler method. For example, an ADD triggers HandlePodAdditions:

switch u.Op {
case kubetypes.ADD:
    klog.V(2).InfoS("SyncLoop ADD", "source", u.Source, "pods", format.Pods(u.Pods))
    handler.HandlePodAdditions(u.Pods)
}

5. HandlePodAdditions performs several steps:

// 1. Sort pods by creation time
sort.Sort(sliceutils.PodsByCreationTime(pods))

// 2. Add pod to podManager (source of truth for desired state)
kl.podManager.AddPod(pod)

// 3. Check if the pod is a static pod
mirrorPod, _ := kl.podManager.GetMirrorPodByPod(pod)

// 4. Dispatch work to the pod worker
kl.dispatchWork(pod, kubetypes.SyncPodCreate, mirrorPod, start)

// 5. Register pod with the probe manager for health checks
kl.probeManager.AddPod(pod)

6. dispatchWork launches an asynchronous worker that calls UpdatePod:

kl.podWorkers.UpdatePod(&UpdatePodOptions{
    Pod:        pod,
    MirrorPod:  mirrorPod,
    UpdateType: syncType,
    OnCompleteFunc: func(err error) {
        if err != nil {
            metrics.PodWorkerDuration.WithLabelValues(syncType.String()).Observe(metrics.SinceInSeconds(start))
        }
    },
})

7. UpdatePod creates a new pod worker (or restarts one) and invokes managePodLoop to process updates:

go func() {
    defer runtime.HandleCrash()
    p.managePodLoop(podUpdates)
}()

8. Inside managePodLoop, each pod update is synchronized via syncPodFn, which eventually calls SyncPod in pkg/kubelet/kuberuntime/kuberuntime_manager.go to align the running pod with the desired state.

// SyncPod syncs the running pod into the desired pod by executing the following steps:
// 1. Compute sandbox and container changes.
// 2. Kill pod sandbox if necessary.
// 3. Kill any containers that should not be running.
// 4. Create sandbox if necessary.
// 5. Create ephemeral containers.
// 6. Create init containers.
// 7. Create normal containers.
func (m *kubeGenericRuntimeManager) SyncPod() {
    podContainerChanges := m.computePodActions(pod, podStatus)
    // ... steps 2‑7 implementation ...
}

9. Additional responsibilities of the pod worker include creating pod data directories, handling volumes, retrieving image pull secrets, and invoking the container runtime's SyncPod method.

The article concludes with a brief invitation to follow the author's public account for more technical content.