How Kubelet’s VolumeManager Orchestrates Async Volume Attach, Mount, and Unmount

VolumeManager runs a set of asynchronous loops that examine the pods scheduled on a node and decide which volumes need to be attached, mounted, unmounted, or detached, then performs those actions.

VolumeManager Interface

type VolumeManager interface {
    // Starts the volume manager and all the asynchronous loops that it controls
    Run(ctx context.Context, sourcesReady config.SourcesReady)

    // WaitForAttachAndMount processes volumes referenced by the given pod and blocks until they are all attached and mounted.
    WaitForAttachAndMount(ctx context.Context, pod *v1.Pod) error

    // WaitForUnmount processes volumes referenced by the given pod and blocks until they are all unmounted.
    WaitForUnmount(ctx context.Context, pod *v1.Pod) error

    // GetMountedVolumesForPod returns a VolumeMap of successfully attached and mounted volumes for the pod.
    GetMountedVolumesForPod(podName types.UniquePodName) container.VolumeMap

    // GetPossiblyMountedVolumesForPod returns a VolumeMap of volumes that are either successfully attached/mounted or “uncertain”.
    GetPossiblyMountedVolumesForPod(podName types.UniquePodName) container.VolumeMap

    // GetExtraSupplementalGroupsForPod returns extra supplemental groups for the pod.
    GetExtraSupplementalGroupsForPod(pod *v1.Pod) []int64

    // GetVolumesInUse returns a list of volumes that implement the volume.Attacher interface and are currently in use.
    GetVolumesInUse() []v1.UniqueVolumeName

    // ReconcilerStatesHasBeenSynced indicates whether the reconciler has synced the actual state at least once.
    ReconcilerStatesHasBeenSynced() bool

    // VolumeIsAttached reports whether the given volume is attached to this node.
    VolumeIsAttached(volumeName v1.UniqueVolumeName) bool

    // MarkVolumesAsReportedInUse marks the given volumes as reported in use.
    MarkVolumesAsReportedInUse(volumesReportedAsInUse []v1.UniqueVolumeName)
}

Starting VolumeManager

VolumeManager is launched from Kubelet.Run:

func (kl *Kubelet) Run(updates <-chan kubetypes.PodUpdate) {
    ...
    go kl.volumeManager.Run(ctx, kl.sourcesReady)
}

Attach & Mount During Pod Sync

When a pod is synchronized, SyncPod calls WaitForAttachAndMount. If the call returns an error, the pod is skipped and an event is recorded.

func (kl *Kubelet) SyncPod(ctx context.Context, updateType kubetypes.SyncPodType, pod, mirrorPod *v1.Pod, podStatus *kubecontainer.PodStatus) (isTerminal bool, err error) {
    ...
    if err := kl.volumeManager.WaitForAttachAndMount(ctx, pod); err != nil {
        if !wait.Interrupted(err) {
            kl.recorder.Eventf(pod, v1.EventTypeWarning, events.FailedMountVolume, "Unable to attach or mount volumes: %v", err)
            klog.ErrorS(err, "Unable to attach or mount volumes for pod; skipping pod", "pod", klog.KObj(pod))
        }
        return false, err
    }
    ...
}

Unmount During Pod Termination

When a pod terminates, SyncTerminatedPod invokes WaitForUnmount. The function blocks until all volumes are unmounted; any error is returned directly, but the surrounding logic still proceeds to delete the pod.

func (kl *Kubelet) SyncTerminatedPod(ctx context.Context, pod *v1.Pod, podStatus *kubecontainer.PodStatus) error {
    ...
    if err := kl.volumeManager.WaitForUnmount(ctx, pod); err != nil {
        return err
    }
    ...
}

Internal volumeManager Struct

type volumeManager struct {
    // volumePluginMgr provides access to registered volume plugins.
    volumePluginMgr *volume.VolumePluginMgr

    // desiredStateOfWorld holds the desired state: which volumes should be attached and which pods reference them.
    desiredStateOfWorld cache.DesiredStateOfWorld

    // actualStateOfWorld holds the actual state: which volumes are attached to the node and which pods they are mounted to.
    actualStateOfWorld cache.ActualStateOfWorld

    // operationExecutor launches asynchronous attach, detach, mount, and unmount operations.
    operationExecutor operationexecutor.OperationExecutor

    // reconciler runs a periodic loop that reconciles desired and actual states.
    reconciler reconciler.Reconciler

    // desiredStateOfWorldPopulator fills desiredStateOfWorld using the kubelet PodManager.
    desiredStateOfWorldPopulator populator.DesiredStateOfWorldPopulator

    // csiMigratedPluginManager tracks CSI migration status of plugins.
    csiMigratedPluginManager csimigration.PluginManager

    // intreeToCSITranslator converts in‑tree volume specs to CSI.
    intreeToCSITranslator csimigration.InTreeToCSITranslator
}

Async Loops Started by volumeManager.run

After volumeManager.run is called, three goroutines are started:

A List‑Watch on CSIDrivers resources.

A “dswp” goroutine that updates desiredStateOfWorld and actualStateOfWorld.

A reconciler goroutine that compares the pod‑desired volume state with the actual state and decides whether to mount or unmount volumes.

Volume Plugin Initialization

The manager obtains plugins via ProbeVolumePlugins. The function registers a fixed set of built‑in plugins (emptyDir, gitRepo, hostPath, nfs, secret, iscsi, downwardAPI, fc, configMap, projected, local, csi) and optionally the image plugin if the feature gate is enabled.

func ProbeVolumePlugins(featureGate featuregate.FeatureGate) ([]volume.VolumePlugin, error) {
    allPlugins := []volume.VolumePlugin{}
    allPlugins = append(allPlugins, emptydir.ProbeVolumePlugins()...)
    allPlugins = append(allPlugins, git_repo.ProbeVolumePlugins()...)
    allPlugins = append(allPlugins, hostpath.ProbeVolumePlugins(volume.VolumeConfig{})...)
    allPlugins = append(allPlugins, nfs.ProbeVolumePlugins(volume.VolumeConfig{})...)
    allPlugins = append(allPlugins, secret.ProbeVolumePlugins()...)
    allPlugins = append(allPlugins, iscsi.ProbeVolumePlugins()...)
    allPlugins = append(allPlugins, downwardapi.ProbeVolumePlugins()...)
    allPlugins = append(allPlugins, fc.ProbeVolumePlugins()...)
    allPlugins = append(allPlugins, configmap.ProbeVolumePlugins()...)
    allPlugins = append(allPlugins, projected.ProbeVolumePlugins()...)
    allPlugins = append(allPlugins, local.ProbeVolumePlugins()...)
    allPlugins = append(allPlugins, csi.ProbeVolumePlugins()...)
    if featureGate.Enabled(features.ImageVolume) {
        allPlugins = append(allPlugins, image.ProbeVolumePlugins()...)
    }
    return allPlugins, nil
}

VolumePlugin Interface

type VolumePlugin interface {
    // Init initializes the plugin; called exactly once before any New* calls.
    Init(host VolumeHost) error
    ...
    // NewMounter creates a Mounter from a volume spec and the enclosing pod.
    NewMounter(spec *Spec, podRef *v1.Pod) (Mounter, error)
    ...
    // NewUnmounter creates an Unmounter from a volume name and pod UID.
    NewUnmounter(name string, podUID types.UID) (Unmounter, error)
    ...
}

Summary of the Mechanism

VolumeManager uses desiredStateOfWorld (populated by the pod manager) and actualStateOfWorld (updated after successful attach/mount operations) to represent the intended and real volume states. The reconciler continuously aligns these two states, triggering attach, mount, detach, or unmount actions via the operationExecutor. This coordination ensures that each pod’s volume lifecycle—attachment, mounting, usage, and eventual unmounting—is correctly managed on the node.