Which Kubernetes Multi‑Cluster Solution Wins? A Deep Dive into KubeFed, Karmada, OCM & Gardener

Background and Core Challenges

Kubernetes multi‑cluster management emerged to address four fundamental problems: (1) deployment model – where to place the control plane and how clusters communicate; (2) cross‑cluster application scheduling – matching workloads to clusters based on policies; (3) extensible application model – supporting custom resources and forward compatibility; and (4) treating a cluster as a first‑class resource for lifecycle management.

Open‑Source Solutions Comparison

KubeFed

KubeFed (now archived) implements a Host Cluster that runs the federation control plane and multiple Member Clusters that host workloads. It introduces the KubeFedCluster CR to join clusters and a FederatedType (e.g., FederatedDeployment) that contains template, placement, and overrides. A typical federated deployment looks like:

apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: test-deployment
spec:
  template:
    ...
  placement:
    clusters:
    - name: cluster1
    - name: cluster2
  overrides:
  - clusterName: cluster2
    clusterOverrides:
    - path: "/spec/replicas"
      value: 5

KubeFed also defines a ReplicaSchedulingPreference CR to distribute replicas across clusters based on static weights. Because the model requires a custom federated CR for every workload, migration from single‑cluster manifests is costly and forward compatibility is limited.

Karmada

Karmada (CNCF sandbox) runs its control plane (API server, scheduler, and per‑cluster controllers) on any Kubernetes cluster or VM. It defines several CRDs:

ResourceTemplate – the original workload (e.g., Deployment).

PropagationPolicy – describes target clusters, replica division, and weighting.

ResourceBinding – stores the binding between the template and the actual resources created in each cluster.

OverridePolicy – per‑cluster customizations.

Work – executes the actual creation.

Example propagation policy for an Nginx Deployment:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: nginx
  placement:
    clusterAffinity:
      clusterNames: [member1, member2]
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
        - targetCluster: member1
          weight: 1
        - targetCluster: member2
          weight: 2

Karmada’s scheduler, de‑scheduler, and taint‑manager continuously adjust ResourceBinding based on cluster health, enabling dynamic, weight‑aware scheduling, taint‑based eviction, and automatic re‑balancing. Network connectivity and cross‑cluster service discovery are provided via Submariner or Istio using the Multi‑Cluster Services API ( ServiceExport / ServiceImport).

Open Cluster Management (OCM)

OCM follows a Hub‑Agent model. The Hub cluster stores desired manifests in ManifestWork objects; each Managed cluster runs a lightweight Klusterlet that pulls the work, applies the resources, and reports status.

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  name: hello-work-demo
  namespace: target-managed-cluster
spec:
  workload:
    manifests:
    - apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: hello
      spec:
        ...

OCM also uses a Placement CR to select clusters (e.g., by label selectors) and a PlacementDecision CR that records the chosen clusters. Dynamic scheduling is performed by the Placement controller, which creates ManifestWorks for the selected clusters.

Gardener

Gardener (SAP) treats Kubernetes as a service (KaaS). It introduces three layers:

Garden Cluster – the control plane that defines Seed, Shoot, and CloudProfile resources.

Seed Cluster – runs the control planes of many Shoot clusters (the “node” layer).

Shoot Cluster – the actual workload clusters.

A Gardenlet runs in each Seed, pulls the desired state from the Garden, and creates the Shoot control plane. This “Kubernetes‑in‑Kubernetes” design ensures uniform Kubernetes versions and configurations across heterogeneous clouds.

Evolution Trends

Multi‑Tenant Isolation

Namespace‑based isolation : Use Kubernetes namespaces, RBAC, ResourceQuota, and NetworkPolicy to separate tenants at the control‑plane level. Secure runtimes (Kata, gVisor, Firecracker) can mitigate kernel‑level attacks.

Cluster‑based isolation : Allocate a dedicated Kubernetes cluster per tenant, providing strong isolation but higher control‑plane overhead.

Virtual‑cluster isolation : Extend the SIG‑Multi‑Cluster proposal to run tenant control planes on shared nodes while keeping data‑plane isolation via VPCs and secure runtimes. Components include a Tenant Operator, Syncer controller, vn‑agent, and an enhanced kube‑proxy.

Future designs may combine virtual clusters with true multi‑cluster management to achieve M:N tenant‑to‑cluster mappings, improving resource utilization while preserving isolation.

Stateful Application Scheduling

Two main techniques are used for migrating stateful workloads across clusters:

Backup/Restore (e.g., Velero) : Velero backs up Kubernetes objects and PersistentVolume snapshots to object storage (S3) and restores them in the target cluster. It defines Backup, BackupController, Restore, and RestoreController CRs.

Checkpoint/Restore (e.g., CRIU) : CRIU captures a running container’s memory, file descriptors, and process state. KEP‑2008 adds a Kubelet API for forensic container checkpointing, available as an alpha feature in Kubernetes v1.25.

A scheduler that needs to handle stateful workloads should detect a special label or annotation, invoke the appropriate backup or checkpoint step, schedule the new replica set, and then restore the state. Additional logic for readiness probes, traffic shifting, and fail‑over is required to maintain service continuity.

Conclusion

The surveyed open‑source projects each address a subset of the core multi‑cluster requirements. Karmada offers the most complete feature set (dynamic scheduling, forward‑compatible application model, and built‑in service export/import). OCM provides a lightweight hub‑agent approach suitable for large‑scale cluster fleets, while Gardener focuses on Kubernetes‑as‑a‑Service with strict version and configuration consistency. KubeFed, although archived, illustrates an early federation model based on static placement.

Current trends point toward hybrid isolation models (virtual clusters) and richer stateful‑application migration mechanisms. Enterprises should evaluate their specific pain points—tenant isolation, scheduling flexibility, or stateful workload migration—before selecting a solution, considering both functional coverage and operational overhead.