In-Place Container Upgrade in Kubernetes: Mechanism and Implementation

Kubernetes treats a Pod as the basic scheduling unit, and all built‑in Workloads (Deployment, StatefulSet, etc.) manage applications at the Pod level. When a Pod contains multiple containers, such as a main business container and one or more Sidecar containers, updating a Sidecar without recreating the entire Pod can significantly reduce downtime.

The article explains why in‑place container upgrade is needed: recreating a whole Pod incurs graceful termination delays, sandbox image pulls, and application start‑up time, which becomes costly for large‑scale deployments or when only a lightweight Sidecar needs an update.

Kubernetes has supported this capability since version 1.5, and the article uses Kubernetes 1.12 source code to illustrate the process.

Key components in the upgrade flow:

The Kubelet’s syncLoopIteration receives a kubetypes.UPDATE event when a Pod spec changes (e.g., container image). It forwards the event to HandlePodUpdates .

HandlePodUpdates dispatches work to podWorker.UpdatePod , which calls Kubelet.syncPod .

KubeGenericRuntimeManager.SyncPod invokes computePodActions to determine what needs to change.

computePodActions examines the Pod sandbox, init containers, and each container’s spec hash. It builds a podActions structure indicating which containers to kill, which to start, and whether the sandbox must be recreated.

pkg/kubelet/kubelet.go:1870
func (kl *Kubelet) syncLoopIteration(configCh <-chan kubetypes.PodUpdate, handler SyncHandler,
    syncCh <-chan time.Time, housekeepingCh <-chan time.Time, plegCh <-chan *pleg.PodLifecycleEvent) bool {
    select {
    case u, open := <-configCh:
        ...
        switch u.Op {
        ...
        case kubetypes.UPDATE:
            glog.V(2).Infof("SyncLoop (UPDATE, %q): %q", u.Source, format.PodsWithDeletionTimestamps(u.Pods))
            handler.HandlePodUpdates(u.Pods)
        ...
    ...
    }
    ...
}

The podActions type includes fields such as KillPod , CreateSandbox , ContainersToKill , and ContainersToStart . Based on these actions, KubeGenericRuntimeManager.SyncPod performs the following steps:

Kill the whole sandbox if required.

Kill individual containers listed in ContainersToKill .

Create a new sandbox if CreateSandbox is true.

Start init containers if needed.

Start regular containers listed in ContainersToStart .

func (m *kubeGenericRuntimeManager) SyncPod(pod *v1.Pod, _ v1.PodStatus, podStatus *kubecontainer.PodStatus, pullSecrets []v1.Secret, backOff *flowcontrol.Backoff) (result kubecontainer.PodSyncResult) {
    // Step 1: Compute sandbox and container changes.
    podContainerChanges := m.computePodActions(pod, podStatus)
    ...
    // Step 2: Kill the pod if the sandbox has changed.
    if podContainerChanges.KillPod {
        ...
    } else {
        // Step 3: kill any running containers in this pod which are not to keep.
        for containerID, containerInfo := range podContainerChanges.ContainersToKill {
            glog.V(3).Infof("Killing unwanted container %q(id=%q) for pod %q", containerInfo.name, containerID, format.Pod(pod))
            killContainerResult := kubecontainer.NewSyncResult(kubecontainer.KillContainer, containerInfo.name)
            result.AddSyncResult(killContainerResult)
            if err := m.killContainer(pod, containerID, containerInfo.name, containerInfo.message, nil); err != nil {
                killContainerResult.Fail(kubecontainer.ErrKillContainer, err.Error())
                glog.Errorf("killContainer %q(id=%q) for pod %q failed: %v", containerInfo.name, containerID, format.Pod(pod), err)
                return
            }
        }
    }
    ...
    // Step 6: start containers in podContainerChanges.ContainersToStart.
    for _, idx := range podContainerChanges.ContainersToStart {
        container := &pod.Spec.Containers[idx]
        startContainerResult := kubecontainer.NewSyncResult(kubecontainer.StartContainer, container.Name)
        result.AddSyncResult(startContainerResult)
        ...
    }
    return
}

Verification is performed by deploying a StatefulSet demo, modifying the image of an nginx container, and observing Kubelet logs such as:

kubelet[1121]: I0412 16:34:28.356083    1121 kubelet.go:1868] SyncLoop (UPDATE, "api"): "web-2_default(2813f459-59cc-11e9-a1f7-525400e7b58a)"
kubelet[1121]: I0412 16:34:28.657836    1121 kuberuntime_manager.go:549] Container "nginx" ({"docker" "8d16517eb4b7b5b84755434eb25c7ab83667bca44318cbbcd89cf8abd232973f"}) of pod web-2_default(...): Container spec hash changed (3176550502 vs 1676109989).. Container will be killed and recreated.
kubelet[1121]: I0412 16:34:28.658529    1121 kuberuntime_container.go:548] Killing container "docker://8d16517eb4b7b5b84755434eb25c7ab83667bca44318cbbcd89cf8abd232973f" with 10 second grace period
...

After the old container is terminated, computePodActions detects that the desired container is not running and schedules startContainer to launch the new image, achieving an in‑place upgrade without recreating the Pod.

The article concludes that understanding this mechanism enables developers to build a CRD/Operator that provides gray‑scale or rolling in‑place upgrades for sidecar containers, which can be exposed as a feature in a custom StatefulSetPlus implementation.

At the end of the article, there is a promotional section inviting readers to a BigData & Alluxio meetup in Chengdu, with links and QR codes. While not part of the technical discussion, it is included in the original source.