How Meituan‑Dianping Scaled Kubernetes to 100k+ Nodes with HULK2.0

Background: Meituan‑Dianping, a leading Chinese lifestyle service platform, experiences extreme traffic spikes during holidays and promotions, demanding highly elastic and available cluster resources while keeping operational costs low.

Evolution of Cluster Management

2013‑2015: Built a traditional virtualized resource delivery system and launched the in‑house scheduler HULK to drive containerization.

2016: Developed a Docker‑based elastic scaling solution, improving deployment speed and resource utilization.

2018: Migrated to Kubernetes for finer‑grained resource management, creating HULK2.0 on top of it.

HULK2.0 Architecture

HULK2.0 abstracts business‑level resource requests via a unified HULK API, decoupling upper‑layer services from the underlying Kubernetes platform while remaining compatible with native Kubernetes APIs.

Key design goals: user‑friendly for business teams, maximal exploitation of Kubernetes scheduling capabilities, and isolation of business logic from low‑level resource details.

Kubernetes Management and Practices

Why Kubernetes? It offers a robust platform rather than a single solution, enabling rapid deployment, dynamic scaling, and advanced resource allocation.

Cluster scale now exceeds 100 000 online instances across multiple regions, with comprehensive monitoring, health alerts, automated inspections, and visual dashboards for nodes, pods, and containers.

Scheduler Optimizations

Performance boost: Upgraded from scheduler v1.6 to v1.10+, achieving >400% throughput improvement.

Pre‑selection interruption: Added alwaysCheckAllPredicates flag to abort further predicate checks once a node fails, reducing computation in large clusters.

Local‑optimal strategy: Limit predicate evaluation to a configurable subset of nodes (e.g., first 100 passing nodes) to achieve near‑optimal placement with far less overhead.

Kubelet Enhancements

Implemented risk‑controlled policies to prevent unwanted pod eviction, added a reusable restart strategy, ensured IP persistence via a custom CNI plugin, and introduced fine‑grained resource controls such as NUMA binding, CPU‑Set allocation, and container‑level limits.

Resource Management and Optimization

Techniques include service profiling (CPU, memory, I/O), affinity/anti‑affinity rules, scenario‑based prioritization, elastic scaling with rule‑based and ML‑driven predictions, and fine‑grained allocation (NUMA, CPU‑Set, shared resources).

Policy optimizations cover affinity, anti‑affinity, application priority, dispersion across hosts/racks/zones, isolation for special workloads, and handling of special hardware (GPU, SSD, NIC).

Online Cluster Optimizations

NUMA binding to reduce cross‑node latency.

CPU‑Set grouping for complementary workloads.

Application staggered deployment based on service profiling to avoid peak contention.

Rescheduling to improve fragmentation and resource utilization.

Interference analysis using monitoring metrics to detect abnormal containers.

Future Directions

Hybrid online‑offline deployment for higher efficiency.

Intelligent, traffic‑aware scheduling to improve SLA.

High‑performance, strongly isolated, and more secure container technologies.

The article concludes with a call for community contributions and notes ongoing research in these areas.