How Tencent Scales Ops Automation for Hundreds of Thousands of Servers

The talk, originally delivered by Liang Ding'an at APMCon 2016, describes how Tencent turned overwhelming operational pressure—over 800 million monthly active users and more than 500 000 physical servers—into an automated, standardized process.

1. What Ops Automation Can and Cannot

Automation is not a panacea; it should target high‑frequency, repeatable tasks that consume 80% of effort while leaving the remaining 20% for human judgment.

In June of the previous year Tencent’s physical machines exceeded 500 000, yet the ops team grew far slower.

Key problems automation aims to solve include outdated documentation, loss of expertise when senior staff leave, hard‑coded IPs, and human errors.

Goal: In large‑scale environments, automatically trigger operations based on monitoring data without human intervention.

2. Dev and Ops: Finding Common Ground

Even before the term DevOps existed, Tencent built the “ZhiYun” platform to align development and operations.

Collaboration is split into four areas, starting with architecture: ops evaluates architectural quality, and developers must follow standardized guidelines.

Traditional “rules‑only” guidelines from other industries often lack enforceability.

Standardization leads to a unified architecture—client, access, logic, and data layers—implemented with framework‑based, component‑based, stateless, and distributed principles.

Frameworks are built in C (not Java) to match internal development habits, and common components (e.g., socket‑based communication) are abstracted for reuse.

3. Technical Details of Ops Automation

After standardization, the workflow is broken into six high‑level steps (23 detailed sub‑steps) covering pre‑deployment, release, testing, gray‑release, and production rollout.

The core of the system is a layered CMDB that stores configuration items, hardware/software specs, and operational procedures, enabling automated deployment and testing.

Standardized device types (CPU‑heavy, memory‑heavy, SSD, etc.) reduce the number of unique objects ops must manage.

Automation proceeds by retrieving module definitions from the CMDB, feeding them to a process engine, and using a C/S command channel to push files and execute commands on target machines.

Security measures include source‑IP whitelisting, role‑based access control, and safeguards against dangerous commands.

Even without full “no‑human‑in‑the‑loop” automation, the system dramatically improves efficiency; for a 500‑machine fleet, ten engineers can manage the entire environment with a few clicks.

4. Real‑World Case

A QQ membership surge triggered an automatic scaling workflow that sent SMS alerts and expanded capacity without manual intervention, demonstrating the practical impact of the seven‑step automation framework.

In summary, by defining clear goals, standardizing assets through a CMDB, integrating development and operations, and orchestrating a robust process pipeline, Tencent achieves high‑efficiency, low‑risk operations at massive scale.