Mastering Multi‑Cluster Management: From Kubernetes Federation v1/v2 to Karmada

Kubernetes claims that a single cluster can support up to 5,000 nodes and 150,000 Pods, but most companies run multiple clusters for reliability, vendor lock‑in avoidance, and fault isolation, requiring a federation layer to manage them uniformly.

Typical federation use cases include:

High availability: deploying applications across clusters to minimize impact of a single cluster failure.

Avoiding vendor lock‑in: spreading workloads across clusters from different providers and migrating when needed.

Fault isolation: using several small clusters instead of one large cluster to contain failures.

Federation v1

The earliest multi‑cluster project, proposed and maintained by the Kubernetes community.

Design started around K8s v1.3, with components and the kubefed CLI released later; it entered beta in v1.6 but was deprecated around v1.11 due to inflexibility and immature APIs.

The basic architecture consists of three components:

Federation API Server: a Kubernetes‑style API server that provides a unified entry point, but only supports resources extended via adapters.

Controller Manager: handles cross‑cluster resource scheduling and state synchronization, similar to kube-controller-manager.

Etcd: stores federation resources.

Creating a federated resource in v1 involves writing all configuration into the resource’s annotations, creating the object in the Federation API Server, and letting the Controller Manager propagate it to member clusters. An example ReplicaSet is shown below:

This architecture has two main problems:

Low flexibility: adding a new resource type requires writing a new adapter and releasing a new version, and objects carry many federation‑specific annotations.

Lack of independent API versioning: some resources are GA in Kubernetes but only beta in Federation v1.

Federation v2

Learning from v1, the community introduced Federation v2, which uses Custom Resource Definitions (CRDs) to implement the entire functionality, eliminating the need for a separate API server. It consists of two components:

Admission webhook for admission control.

Controller manager that processes custom resources and coordinates state across clusters.

Creating a federated resource in v2 follows a similar high‑level flow, but the configuration is expressed via CRDs. The process diagram is shown below:

The federated resource is stored in the host cluster’s API server; the controller manager then distributes it to member clusters according to the rules defined in the federated resource object.

Logically, Federation v2 is split into two parts: configuration and propagation. Configuration includes Cluster Configuration and Type Configuration.

Cluster Configuration

Stores API authentication information of member clusters. Use kubefedctl join / kubefedctl unjoin to add or remove clusters. Upon successful join, a KubeFedCluster CR is created to hold endpoint, CA bundle, token, etc. The controller manager uses this information to talk to member clusters.

apiVersion: core.kubefed.io/v1beta1
kind: KubeFedCluster
metadata:
  creationTimestamp: "2019-10-24T08:05:38Z"
  generation: 1
  name: cluster1
  namespace: kube-federation-system
spec:
  apiEndpoint: https://172.16.200.1:6443
  caBundle: LS....Qo=
  secretRef:
    name: cluster1-shb2x
status:
  conditions:
  - lastProbeTime: "2019-10-28T06:25:58Z"
    lastTransitionTime: "2019-10-28T05:13:47Z"
    message: /healthz responded with ok
    reason: ClusterReady
    status: "True"
    type: Ready
  region: ""

Type Configuration

Defines which Kubernetes API resources are managed by the federation. For example, to federate ConfigMap, you first create a federated CRD FederatedConfigMap in the host cluster, then create a FederatedTypeConfig named configmaps that maps the native ConfigMap to the federated type.

apiVersion: core.kubefed.k8s.io/v1beta1
kind: FederatedTypeConfig
metadata:
  name: configmaps
  namespace: kube-federation-system
spec:
  federatedType:
    group: types.kubefed.k8s.io
    kind: FederatedConfigMap
    pluralName: federatedconfigmaps
    scope: Namespaced
    version: v1beta1
  propagation: Enabled
  targetType:
    kind: ConfigMap
    pluralName: configmaps
    scope: Namespaced
    version: v1

Federated Resource CRD

The key CRD that describes a federated resource. It contains three sections:

Templates – the actual resource definition.

Placement – which clusters the resource should be deployed to.

Overrides – per‑cluster customizations.

Example for a federated Deployment:

apiVersion: types.kubefed.k8s.io/v1beta1
kind: FederatedDeployment
metadata:
  name: test-deployment
  namespace: test-namespace
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      ...
  placement:
    clusters:
    - name: cluster2
    - name: cluster1
  overrides:
  - clusterName: cluster2
    clusterOverrides:
    - path: spec.replicas
      value: 5

Federated resources are created with kubefedctl enable <resource type> or by manually applying the CRD.

Karmada

Karmada is Huawei’s open‑source multi‑cloud orchestration project, inheriting concepts from both Federation v1 and v2.

Karmada consists of three core components:

Karmada API Server – a standard Kubernetes API server backed by its own etcd, storing federated resources.

Karmada Controller Manager – a set of controllers that watch the API server and communicate with member clusters.

Karmada Scheduler – provides advanced multi‑cluster scheduling policies.

Resources are created in the Karmada API server; the controller manager then propagates them to member clusters according to a dedicated PropagationPolicy CR, eliminating the need for the separate federated‑resource CRDs used in v2.

Cluster

The Cluster resource records the endpoint, CA bundle, and token of a member cluster. Karmada supports two sync modes: Push – the host cluster actively pushes updates to the member. Pull – a karmada-agent runs on the member cluster, pulling status back to the host.

spec:
  apiEndpoint: https://172.31.165.66:55428
  secretRef:
    name: member1
    namespace: karmada-cluster
  syncMode: Push

Propagation Policy

Defines how a native Kubernetes resource (e.g., a Deployment) is distributed to member clusters. The resource template is a regular Kubernetes object, and the policy specifies target clusters, replica distribution, and optional selectors.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: nginx
  placement:
    clusterAffinity:
      clusterNames:
      - member1
      - member2
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
        - targetCluster:
            clusterNames:
            - member1
          weight: 1
        - targetCluster:
            clusterNames:
            - member2
          weight: 1

After applying the resources, the Deployment appears in the member clusters, not in the host cluster where the API server resides.

Override Policy

Allows per‑cluster customizations by defining an OverridePolicy CR. For example, you can replace the container image in a specific region or add annotations.

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: example-override
  namespace: default
spec:
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: nginx
  targetCluster:
    clusterNames:
    - member1
    labelSelector:
      matchLabels:
        failuredomain.kubernetes.io/region: dc1
  overriders:
    plaintext:
    - path: /spec/template/spec/containers/0/image
      operator: replace
      value: 'dc-1.registry.io/nginx:1.17.0-alpine'
    - path: /metadata/annotations
      operator: add
      value:
        foo: bar

For a complete reference, see the API definitions of Placement, ReplicaSchedulingPreference, and related CRDs.