Tencent Games’ 3‑Year Journey of Kubernetes Adoption and Optimization for Large‑Scale Online Gaming

Three years ago Tencent Games began exploring containerization to improve resource utilization and standardize deployment with Docker images. After evaluating alternatives such as Shipyard and Fig, the team adopted the early open‑source Kubernetes (v0.4) and heavily customized it to become the core of their massive game‑container scheduling platform.

The platform supports two primary deployment scenarios: a lightweight VM mode where each container hosts multiple services and requires a dedicated internal IP, and a micro‑service mode where each container runs a single service and uses virtual IPs. Over 20,000 containers across more than 200 games are now managed in data centers worldwide.

Key customizations to the scheduler include network resource allocation (binding IPs to minions), CPU binding to physical cores to avoid contention, NUMA‑aware placement, and disk quota management using Ceph. Anti‑affinity policies and IDC‑aware placement ensure high availability and reduced impact of node failures.

Network performance is addressed with two solutions: a Bridge + VLAN setup for general workloads and an SR‑IOV based approach for latency‑sensitive games, achieving near‑bare‑metal throughput. The team also refined Docker’s NAT limitations by assigning each pod a unique internal IP.

Resource quotas are enforced via cgroups for CPU/Memory and XFS quotas for disk, with dynamic vertical scaling to handle fluctuating player loads without downtime. Buffered‑IO throttling is supported through a custom kernel patch.

Monitoring transitioned from cAdvisor to a solution built on lxcfs and an internal agent, exposing container‑level metrics such as /proc/meminfo through a FUSE‑based virtual proc filesystem.

Persistent storage evolved from simple HostPath mounts to a Ceph‑backed solution integrated into Kubernetes, providing reliable data durability for stateful game services.

In 2018 the platform migrated to Kubernetes 1.2 to support the first micro‑service‑based game (e.g., "极品飞车 Online"). The architecture now relies on overlay networking (Flannel) for pod‑to‑pod traffic, SR‑IOV for high‑performance external traffic, and NAT for internal corporate access, with HAProxy handling internet ingress.

Additional enhancements include replication controllers for high availability, token‑based authentication, ABAC authorization, admission controllers, ServiceAccount usage, ConfigMap/Secret configuration management, and DNS‑based service discovery via SkyDNS.

The team also operates a dual‑version Docker registry (v1/v2) behind Nginx, implements regional mirror registries and plans P2P distribution to improve image pull performance, and open‑sources the SR‑IOV CNI plugin for community use.