How Ctrip Scaled Karmada to Handle 200 GB Memory Peaks and Hundreds of Thousands of Pods

Why Ctrip moved from Kubefed to Karmada

Early federation at Ctrip used Kubefed for cross‑cluster resource sync, but as workloads grew the model exposed three limits: extra adaptation cost for federated objects, tight coupling of platform policies with business configs, and lack of a mechanism to express instance‑distribution weights for local HPA scaling.

Karmada was adopted because it keeps native Kubernetes objects unchanged and separates business resources (Deployment, Service, ConfigMap, HPA) from platform policies (PropagationPolicy, OverridePolicy), allowing weight‑driven distribution and independent local scaling.

Production deployment of Karmada

Karmada now powers multi‑cluster high‑availability, cross‑cluster migration, and baseline component delivery. The core principle is to keep low‑frequency, global capabilities in the federation layer while retaining high‑frequency, local capabilities (real‑time HPA, traffic switching) in member clusters.

For high‑availability, each stateless service runs in multiple clusters with independent HPA. When a cluster fails, traffic is removed from the failed instances and other clusters automatically scale up, ensuring rapid self‑healing without relying on the federation control plane.

Weight‑driven migration uses a custom fed‑hpa‑controller and a Rebalance state machine. Administrators set target weight ratios in PropagationPolicy; the controller aggregates replica counts across clusters and drives gradual, gray‑scale migration while respecting HPA constraints. Non‑HPA workloads use a similar gray‑scale controller.

This mechanism has already moved over hundreds of thousands of Pods.

Baseline component delivery acceleration

Before Karmada, initializing a new cluster required days of manual coordination across teams. Karmada centralizes baseline component distribution via a Trip Cluster Components Controller that respects component dependencies and order, reducing initialization time from days to hours.

Scaling challenges and optimizations

As resource count grew to hundreds of thousands, the Karmada control plane faced three pressure points: high‑frequency state sync, controller startup cost, and memory usage.

High‑frequency sync : HPA status updates turned Work resources into hot‑update objects, causing hundreds of updates per second, etcd size growth, write latency, and 409 conflicts. Optimizations included removing Work for runtime state, lowering HPA update frequency, batching watch events, and upgrading etcd to 3.4.

Startup cost : Large object lists slowed informer cache sync, causing long List operations, watch cache “too old” errors, and prolonged reconciliation. Mitigations added pre‑loading informers, extending watch cache timeout (available from Kubernetes v1.33), prioritizing new events, and skipping unchanged objects via deep‑equal checks.

Memory pressure : The controller manager stored distributed resources, control‑plane objects, member‑cluster informer caches, and deserialized objects, leading to a 200 GB memory peak. Tests on Kubernetes 1.36 showed a ~15% reduction using watch‑list optimizations.

Future directions include enabling the CBORServingAndStorage feature gate, using metadata informers for cold data, typed informers for stable objects, byte‑caches for large objects, and controller sharding to limit per‑process memory.

Policy permission boundaries

PropagationPolicy and ClusterPropagationPolicy can unintentionally affect other teams' resources in a shared control plane. Native RBAC lacks fine‑grained constraints, so Ctrip introduced OPA admission checks to validate policy creators, target labels, and selector scopes.

Future work

Ctrip plans to retire the remaining Kubefed stack, consolidate more infrastructure governance into Karmada, and continue improving control‑plane stability, cache construction, memory limits, and controller partitioning.

Overall, the key lesson is to keep the federation layer focused on low‑frequency, global duties while preserving local autonomy for high‑frequency, runtime‑critical functions.