Why Make PersistentVolume Node Affinity Mutable? Benefits and Risks in Kubernetes

Kubernetes v1.1 introduced the concept of PersistentVolume (PV) node affinity, which restricts which nodes can access a volume. Initially immutable, the field became mutable in the v1.35 alpha release, opening the door to more flexible online volume management.

Why make node affinity mutable?

Stateful workloads cannot be recreated without data loss, unlike stateless Deployments. Modern storage providers now offer regional disks and support online migration from zonal to regional volumes without interrupting workloads. These changes are expressed via the VolumeAttributesClass API (released in v1.34). However, even after a volume migrates, the PV still records the old node‑affinity constraints, preventing Pods from being scheduled to nodes in the new zone. Making the spec.nodeAffinity field mutable allows administrators to update these constraints to match the new storage location.

Typical updates

Example: changing affinity from a specific zone to a region.

spec:
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - us-east1-b

Modified to:

spec:
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/region
          operator: In
          values:
          - us-east1

Another scenario involves storage‑provider‑specific disk types. When a provider releases a new generation of disks that older nodes cannot mount, the PV affinity must be updated accordingly.

spec:
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: provider.com/disktype.gen1
          operator: In
          values:
          - available

Modified to:

spec:
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: provider.com/disktype.gen2
          operator: In
          values:
          - available

Enabling the feature

The mutable node affinity feature is in Alpha, disabled by default, and gated by the MutablePVNodeAffinity feature gate. Administrators must enable the gate on the API server, have appropriate RBAC permissions, and ensure the underlying storage is updated on the provider side before editing the PV.

Race condition between updates and scheduling

Relaxing node affinity is safe, but tightening it can create a race: the scheduler may cache the old PV state and schedule a Pod to a node that no longer satisfies the new affinity, leaving the Pod stuck in ContainerCreating. A proposed mitigation is to let the kubelet fail the Pod start if the PV affinity is violated, though this is not yet implemented.

Future integration with CSI

Currently, both the PV affinity change and the underlying volume update are manual tasks for cluster admins. The long‑term goal is to integrate this workflow with VolumeAttributesClass and CSI, allowing non‑privileged users to trigger storage‑side updates via a PVC change, with the PV node affinity automatically synchronized.