How OpenKruise SidecarSet Enables Zero‑Downtime Hot‑Upgrade for Service Mesh Containers

OpenKruise SidecarSet Hot‑Upgrade for Service Mesh

OpenKruise v0.9.0 adds a Service Mesh‑focused hot‑upgrade capability to the SidecarSet workload. It enables seamless replacement of mesh sidecar containers (e.g., Envoy) without stopping the pod’s traffic.

Architecture

{Sidecar.name}-1 : the active sidecar (e.g., envoy:1.16.0).

{Sidecar.name}-2 : an empty placeholder container ( HotUpgradeEmptyImage) that does nothing during normal operation.

When a pod is created, the SidecarSet webhook injects both containers. The placeholder shares the same configuration (commands, lifecycle, probes) except for the image.

Hot‑Upgrade Process

Upgrade : replace the empty container’s image with the new sidecar version (e.g., envoy:1.17.0).

Migration : a postStart hook runs a migration script inside the sidecar to perform mesh‑specific actions.

Reset : after the mesh service is upgraded, replace the old sidecar with the empty container again.

Repeating these three steps allows multiple hot upgrades.

Coordination via Environment Variables

The sidecars receive two environment variables injected by SidecarSet:

SIDECARSET_VERSION
SIDECARSET_VERSION_ALT

The Go helper isHotUpgradeProcess() reads these variables and returns:

a boolean indicating whether a hot‑upgrade is in progress, and

a boolean indicating whether the current sidecar is newer than its counterpart.

ListenFD Transfer Using Unix Domain Socket

To avoid connection loss, the old sidecar transfers its listening file descriptor (ListenFD) to the new sidecar via a Unix Domain Socket (UDS). The simplified flow is:

Obtain the FD from the old sidecar’s *net.TCPListener.

Send the FD over a UDS ( /dev/shm/migrate.sock).

Close the old listener to stop accepting new requests (drain phase).

The new sidecar receives the FD, reconstructs a *net.TCPListener, and resumes serving traffic.

// Example of sending the FD from the old sidecar
fd := tcpLn.File().Fd()
rights := syscall.UnixRights(int(fd))
addr, _ := net.ResolveUnixAddr("unix", "/dev/shm/migrate.sock")
uds, _ := net.DialUnix("unix", nil, addr)
uds.WriteMsgUnix(nil, rights, nil)
// Close old listener
tcpLn.Close()

// Example of receiving the FD in the new sidecar
addr, _ := net.ResolveUnixAddr("unix", "/dev/shm/migrate.sock")
ln, _ := net.ListenUnix("unix", addr)
conn, _ := ln.AcceptUnix()
buf := make([]byte, 32)
oob := make([]byte, 32)
_, oobn, _, _, _ := conn.ReadMsgUnix(buf, oob)
scms, _ := syscall.ParseSocketControlMessage(oob[:oobn])
if len(scms) > 0 {
    fds, _ := syscall.ParseUnixRights(&scms[0])
    f := os.NewFile(uintptr(fds[0]), "")
    listener, _ := net.FileListener(f)
    http.Serve(listener.(*net.TCPListener), serveMux)
}

SidecarSet Specification Example

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: hotupgrade-sidecarset
spec:
  selector:
    matchLabels:
      app: hotupgrade
  containers:
  - name: sidecar
    image: openkruise/hotupgrade-sidecarv1
    imagePullPolicy: Always
    lifecycle:
      postStart:
        exec:
          command: [/migrate.sh]
    upgradeStrategy:
      upgradeType: HotUpgrade
    hotUpgradeEmptyImage: openkruise/hotupgrade-empty

Real‑World Adoption

Alibaba Cloud Service Mesh (ASM) uses this hot‑upgrade mechanism to provide a beta data‑plane sidecar upgrade feature, allowing users to upgrade mesh proxies without service impact.

References

Hot‑Upgrade demo repository: https://github.com/openkruise/samples OpenKruise source code:

https://github.com/openkruise/kruise