From Manual Ops to Automated Cloud: A 7‑Year Journey of a Game Ops Team

Introduction

Over more than seven years the operations team grew from a 2‑3 person squad to a comprehensive functional center handling systems, development, and a range of self‑built technologies such as a CDN system, network analysis and alert platform, and an automation platform.

The team now manages tens of thousands of servers, supports over a hundred game projects, runs a self‑built CDN with T‑level capacity, and handles game servers that sustain ten‑thousand‑player online loads.

1. Primitive Era – “Guard” (2008‑2009)

In the early days operations were manual, relying on DNS round‑robin load balancing, with no automation, high disk‑failure rates, immature monitoring, and ad‑hoc coordination. Leadership often performed hands‑on troubleshooting, setting a culture of leading by example.

2. Stone Age – “Attack” (2010‑2012)

Adoption of Linux open‑source tools (Nagios, Cacti, Puppet, Salt, OSSEC, etc.) brought systematic monitoring and automation. The booming page‑game market demanded rapid server provisioning, leading to the development of a cluster‑based opening architecture that consolidated multiple game instances onto high‑spec physical servers, improving load performance, reducing hardware costs, and eliminating redundant “one‑server‑per‑instance” setups.

Key benefits of the cluster opening architecture included higher online load capacity, reduced server pressure, faster opening efficiency, lower maintenance effort, high replicability, and significant cost savings.

3. Thawing Era – “Transition” (2013‑2015)

The rise of mobile gaming forced a shift in architecture. An operations center was established, introducing SOPs, D/O separation, and a suite of automation platforms such as the “Blue Sea” system, a unified CDN management platform, and a public‑cloud solution built on OpenStack and Ceph.

Big‑data log processing was achieved with an ELK + Redis stack, handling up to 70 K events per second and billions of daily log entries, enabling detailed analytics for CDN and cloud services.

Public‑cloud deployment faced hardware constraints (1 Gbps network, SAS disks) but achieved impressive write performance through dual‑NIC bonding and RAID‑0 on Ceph OSD nodes, demonstrating the feasibility of a cost‑effective, highly available cloud infrastructure.

4. Rebirth Era – “New” (2016‑Present)

Operations has moved toward service‑oriented, platform‑based delivery, emphasizing business impact, self‑service capabilities, and continuous automation. The focus is on converting technical expertise into production value, supporting rapid deployment, fault tolerance, and scalable resource management across diverse game projects.

Future directions include deeper platformization, enhanced soft skills, and innovative service models that position operations as a strategic business partner rather than a behind‑the‑scenes support function.