Mixed Version Proxy in Kubernetes 1.28: Overview, Operation, and Enablement

This article introduces the mixed version proxy, a new alpha feature in Kubernetes 1.28 that allows clusters running multiple versions of the API server to correctly serve HTTP requests for resources. It is especially useful during cluster upgrades or when configuring the control plane at runtime.

What problem does this feature solve? During upgrades, different kube‑apiserver versions may serve different built‑in resource groups, versions, or resources. Requests can be routed to an apiserver that does not recognize the requested resource, resulting in incorrect 404 errors that may block namespace deletion or garbage collection.

How does it work?

The new filter added to the apiserver aggregation layer checks whether a request targets a group/version/resource unknown to the current apiserver (using the existing StorageVersion API). If so, it proxies the request to one of the apiservers listed in the ServerStorageVersion object. If the chosen peer cannot respond, a 503 Service Unavailable is returned.

To avoid infinite proxy loops, the original apiserver adds a new HTTP header (introduced in v1.28): X-Kubernetes-APIServer-Rerouted: true . When a peer sees this header set to true, it knows not to proxy the request again.

Network locations for peer apiservers can be supplied via the --advertise-address and --peer-advertise-ip flags, allowing correct routing even when direct communication between peers is restricted.

How to enable the feature?

Download Kubernetes source version v1.28.0 or newer.

Enable the feature gate on the kube‑apiserver: --feature-gates=UnknownVersionInteroperabilityProxy=true

Provide a CA bundle for peer verification using --peer-ca-file (required).

Supply the correct IP and port of the local apiserver for peers using --peer-advertise-ip , --peer-advertise-port , --advertise-address , or --bind-address as appropriate.

Additional considerations

Currently, the proxy is used only when it is known to be necessary. Future beta releases will add merging of discovery requests across all apiservers and an outbound dialer for peer network connections.

Where to learn more?

Read the mixed version proxy documentation and KEP‑4020 (Unknown Version Interoperability Proxy) for detailed design and implementation notes.

How to get involved?

Join the SIG API Machinery Slack channel (#sig-api-machinery) or the mailing list. Thanks to contributors Daniel Smith, Han Kang, Joe Betz, Jordan Liggit, Antonio Ojea, David Eads, and Ben Luddy for their work.