Exploring Container Federation: Multi‑Cluster Management with FOOT V3.5

Introduction

As cloud‑native technologies rapidly mature, enterprises face the limitation of single‑cluster deployments and the growing need to manage resources across multiple container clusters. Traditional tools often handle only one cluster, prompting the emergence of multi‑cluster management solutions.

Business Pain Points

Centralized Management: Provide a single platform to onboard, configure, monitor, and maintain many clusters.

Resource Scheduling & Load Balancing: Dynamically allocate workloads based on each cluster’s capacity.

Unified Security: Apply consistent access control, authentication, and encryption across clusters.

Fault Recovery & Disaster Backup: Automatically migrate workloads when a cluster fails.

Cross‑Cluster Application Deployment: Abstract an application model that can be deployed uniformly to all clusters.

Open‑Source Federation Projects

Several projects address multi‑cluster needs:

Federation V2 (KubeFed): A Kubernetes SIG Multi‑Cluster architecture for federating independent clusters.

Rancher: Offers a UI‑driven multi‑cluster management platform.

Karmada: An open‑source CNCF project enabling cross‑cluster scheduling without modifying applications.

OCM (Open Cluster Management): Provides a sandbox‑level framework for resource orchestration across clouds.

Docker Swarm and Mesos also support multi‑cluster scenarios but lag behind in features and community support.

FOOT V3.5 Platform Overview

The FOOT platform builds on OCM and Karmada, delivering a fully autonomous solution that hides cloud‑specific differences behind a unified Kubernetes API. It introduces a lightweight hub cluster that hosts control‑plane components and manages member clusters via push or pull registration.

Push Mode

In push mode, FOOT creates a service account in each member cluster, grants it cluster‑role permissions, and uses the control‑plane to push resources directly to the cluster’s API server.

Pull Mode

In pull mode, a FOOT agent runs inside the member cluster, pulls resources from the hub, and registers using the script

hack/deploy-foot-agent.sh ~/.kube/foot.config foot-apiserver ~/.kube/foot.config member2

. The agent creates a service account, cluster‑role, and a secret containing the hub’s kubeconfig, then runs a controller manager to monitor cluster health.

Application Lifecycle Management

FOOT uses an ApplicationPolicy Controller to govern two policy types:

PropagationPolicy: Defines target clusters and replica distribution for workloads.

OverridePolicy: Handles cluster‑specific overrides such as image registries, commands, and arguments.

These policies enable multi‑cluster deployment, canary releases, blue‑green strategies, and automatic failover.

APISIX‑Based Multi‑Cluster Gateway

Integrating Apache APISIX provides dynamic configuration, advanced load‑balancing (Round Robin, Least‑Conn, EWMA, etc.), gray‑release capabilities, and extensible plugins for authentication and observability, addressing the shortcomings of NodePort, LoadBalancer, and standard Ingress.

Ceph‑Based Distributed Storage

FOOT adopts Ceph with a custom CRUSH algorithm to evenly distribute data across cluster nodes, offering block, file, and object storage. The software‑defined storage layer supports high‑performance, secure, and scalable storage for stateful cloud‑native workloads.

Architecture Diagram

Key Advantages

Heterogeneous resource integration across on‑prem, public clouds, and edge clusters.

Cross‑cluster autonomous application deployment with rich scheduling policies.

Robust service distribution via APISIX, supporting multi‑protocol routing and dynamic rule loading.

Reliable distributed storage backed by Ceph, delivering unified block, file, and object services.

Future Outlook

Upcoming developments will blend AI for intelligent scaling and fault recovery, extend edge‑cloud collaboration, and strengthen cloud‑native security mechanisms such as container and network hardening.