Kubernetes Mixed Version Proxy Reaches Beta in v1.36, Strengthening Cluster Upgrade Reliability

Upgrading a Kubernetes cluster is a routine but delicate operation, especially in highly available control planes where multiple API servers may run different versions. A single misrouted request can cause cascading issues. Since version 1.28, the Mixed Version Proxy (MVP) has worked to address this as an alpha feature. Now, with Kubernetes v1.36, MVP graduates to beta and is enabled by default, bringing significant architectural improvements and enhanced reliability.

The Problem of Mixed-Version Control Planes

During a rolling upgrade of a multi-master cluster, API servers temporarily run heterogeneous versions. Each server may serve a different set of APIs—newer servers might support resource versions that older ones do not. Without a proxy mechanism, if a client request hits an older API server that doesn't yet know about a resource (e.g., an API version introduced in the upgrade), that server returns a 404 Not Found. This response is technically incorrect: the resource exists in the cluster, just not on that server. The consequences can be severe, including:

• Mistaken garbage collection of valid resources
• Blocked namespace deletion due to phantom missing resources
• Application failures and confusing debugging

The Mixed Version Proxy was designed to eliminate these false negatives by intelligently routing requests to a peer API server that can serve them.

How the Mixed Version Proxy Works

When a request arrives at an API server that cannot serve the resource locally, MVP steps in. The server consults a discovery cache to identify a capable peer, then proxies the request—adding a special x-kubernetes-peer-proxied header to prevent loops. The peer processes the request and returns the response, which is forwarded to the original client. This transparent redirection ensures clients never see an incorrect 404 and that the upgrade proceeds smoothly.

1. Client sends a request (e.g., for API v2) to an older API Server A.
2. Server A determines it cannot serve locally.
3. Server A looks up a capable peer in its discovery cache.
4. Server A proxies the request to Server B (newer).
5. Server B processes and returns the response.
6. Server A forwards the response to the client.

Key Improvements from Alpha to Beta

The original alpha implementation proved the concept but relied on older mechanisms and had gaps. Beta brings two crucial enhancements.

From StorageVersion API to Aggregated Discovery

In alpha, API servers used the StorageVersion API to learn which peers served which resources. While functional, this approach had a major limitation: the StorageVersion API was not supported for Custom Resource Definitions (CRDs) or aggregated APIs. This meant MVP could not properly proxy requests for those resources, limiting its usefulness for many real-world clusters.

For beta, the team replaced the StorageVersion dependency with Aggregated Discovery. Now, API servers dynamically gather discovery data from peers, understanding the full set of APIs each server can serve—including CRDs and aggregated API servers. This makes the proxy universal and far more reliable.

The Missing Piece: Peer-Aggregated Discovery

In the alpha release, a significant gap remained: while resource requests could be proxied, discovery requests (e.g., GET /api) only returned APIs known to the local server. A client that queried the wrong server would still receive an incomplete list, potentially missing newer API versions. This undermined the value of the proxy.

Starting in v1.36, the Mixed Version Proxy also handles discovery requests. It now aggregates discovery information from all peer API servers and returns a complete, version-aware response. This ensures that clients always see the full API surface of the cluster, regardless of which server they contact.

What This Means for Cluster Operators

With MVP now beta and enabled by default in fresh installs and upgrades to v1.36, operators get immediate benefits:

• No configuration required—the proxy activates automatically for control plane components.
• Safer upgrades—incorrect 404 errors during rollouts are virtually eliminated.
• Better CRD support—thanks to aggregated discovery, CRD resources are now proxied correctly.
• Transparent operation—clients and controllers experience no change; the proxy works behind the scenes.

If you are managing a cluster with multiple API server instances, you should verify that your monitoring no longer sees spurious 404 errors during upgrades. The MVP also lays groundwork for more advanced multi-version control plane features in the future.

Looking Ahead

The Kubernetes community continues to refine the Mixed Version Proxy. The beta stabilizes the core behavior, but further work may include performance optimizations, support for watch requests, and tighter integration with the API priority and fairness system. The move to beta signals confidence that the feature is production-ready for most scenarios.

For more details, refer to the official Mixed Version Proxy documentation and the KEP.

The Mixed Version Proxy is a prime example of how the Kubernetes project evolves to handle real-world operational challenges. With v1.36, it becomes a default safeguard for anyone running highly available clusters.