How We Built a Highly Available Kubernetes Platform for Multi‑Cluster Deployments

Background

We chose Kubernetes because it supports almost all container workloads—stateless, stateful, batch, and daemonset—and has become the de‑facto standard for container orchestration. It also improves resource utilization, development, testing, operations and DevOps efficiency.

Solution

Overall Architecture

The platform is a web‑based operation system that handles most daily developer tasks such as application release, scaling, rollback, blue‑green deployment, CI/CD pipelines, and log/monitoring views.

Cluster Deployment

To achieve high availability, the platform deploys multiple IDC‑level Kubernetes clusters. An application can be deployed to different IDC clusters simultaneously, and two clusters can run in the same IDC to avoid cross‑IDC latency and scheduler performance issues. Both self‑built clusters and cloud‑provider clusters (e.g., UCloud UK8S) are supported. A custom k8s‑sync component synchronizes container IPs to a unified ingress; if synchronization fails in one cluster, the deployment aborts without affecting other clusters.

Master High Availability

The master layer must be highly available. etcd , the only stateful component, runs on an independent etcd cluster (3 or 5 nodes) that can be upgraded autonomously via CoreOS tools. kube‑apiserver is stateless and placed behind a load balancer. kube‑controller‑manager and kube‑scheduler achieve HA through multiple instances and leader election via locked endpoints.

Logging

Logs written to stdout / stderr are collected by Filebeat and sent to Kafka. Java applications use a custom agent‑based solution that was originally built for VMs and retained for containers.

Cluster Monitoring

Node monitoring continues to use Open‑Falcon. For containers we collect metrics with cAdvisor and kube‑state‑metrics. Core Kubernetes components expose metrics that Prometheus scrapes and visualizes in Grafana and the operation platform.

Application Monitoring

Developers see CPU, memory, disk I/O and network I/O metrics via cAdvisor and kube‑state‑metrics, plus alerts for container restarts, crashes, image pull failures and orphaned Pods.

Service Exposure

We evaluated Traefik but adopted an internal ingress layer (yz7) and built k8s‑sync to watch Endpoints and sync IPs to yz7. RPC services use macvlan to keep compatibility with the VM era.

Image Building

Images consist of OS, runtime and application layers. Each application type follows strict port conventions. Python and Node.js apps only need an app.yaml at the repository root, for example:

stack: youzanyun-centos6
runtime: python-2.7
entrypoint: gunicorn -c gunicorn_config.py wsgi:application

Labels

Pods are labeled with application name, cluster, environment, IDC, and deployment stage (gray/blue‑green) to facilitate management and future affinity/anti‑affinity rules.

Lifecycle Hooks

We use Kubernetes postStart and preStop hooks. On start, preload and online scripts perform health checks before the application goes live. On termination, offline and stop scripts execute the reverse sequence.

Continuous Delivery

CI/CD is realized through project environments; each standard environment is deployed to a separate Kubernetes cluster and isolated by namespace.

Multi‑Cluster Management

Our production includes self‑built clusters and UCloud UK8S clusters. A custom management platform lets administrators create clusters, add or remove nodes, tag nodes, and monitor resource usage.

Problems Encountered

CPU core count reported incorrectly for Java containers; resolved by upgrading JDK after an initial lxcfs hack.

CrashLoopBackOff during releases; a debug mode disables health checks and lifecycle hooks to allow deployment, and an isolation feature tags problematic Pods to detach them from Deployments for post‑mortem analysis.

Side‑car container dependencies; we introduced a “rich container” pattern to control start order, and the community is working on a formal side‑car solution.

Future Outlook

We plan to adopt Operators to streamline Kubernetes usage, enable HPA and VPA for automated scaling, and improve scheduling granularity to increase cluster utilization.