How Guangdong Mobile Scaled AIOps: From Manual Ops to Intelligent Automation

Guangdong Mobile IT System and Operations Development

Guangdong Mobile’s IT systems are divided into four domains: B (business support), M (management support), O (network management), and S (industry solutions). The AIOps applications focus on the B and M domains.

The B domain handles massive business support workloads (over 1.3 billion customers, 2.5 million daily transactions, 160 billion daily call records) with more than 50 core modules and 2,000+ host nodes, spanning mainframes, virtual machines, and containers. The M domain supports management tools (OA, email, HR, finance) across 70+ systems using diverse stacks (iOS, Android, Java) and databases (MySQL, Oracle).

Operational capability has progressed through several milestones:

2010 – Process‑standardized operations.

2018 – Platformization and automation of IT network management.

2020 – Orchestration and intelligence (smart anomaly detection, root‑cause analysis, knowledge base).

2023 – AIOps becomes a major part of daily operations.

2025 – Goal to achieve extensive automation and unattended operations.

AIOps Implementation and Practice

Research began in 2019 with scenario pre‑studies; by 2020, 27 scenarios (anomaly detection, root‑cause analysis, alarm convergence, knowledge graph) were piloted, establishing a framework of compute + algorithm + knowledge graph.

2021 expanded objects (middleware, databases, network), scenarios (cost management, resource optimization), and integration, forming a compute + algorithm + data framework. In 2022, a five‑dimensional model (perception, analysis, decision, execution, knowledge update) was applied, achieving L3 maturity and adding a knowledge‑graph layer.

Four major operational pain points were addressed with AIOps capabilities:

Smart anomaly detection – covering 20 resource types, 100+ metrics, 90+ systems, 13 000+ monitoring points, achieving 98% recall and 93% precision.

Smart change assessment – pre‑change risk evaluation using change‑order data and business graph, and post‑change automated assessment, reducing 28 change‑related incidents in 2022.

Smart fault diagnosis – two‑generation models using logs, alarms, KPIs, traces, and assets, delivering 89% diagnosis accuracy, cutting operation costs by 80% and improving fault handling efficiency by 62%.

Intelligent operation robot – providing smart Q&A, ticket recommendation, and automated remediation, reducing monthly tickets by 15 000 and saving millions in labor costs.

Overall AIOps adoption reduced annual labor costs by over ten million, cut alarm handling time by 75%, halved average fault detection time, and shortened IT complaint processing by 93.5%.

Promotion Methods and Platform Construction

The “TRL” promotion framework (Target, Review, Limit) emphasizes clear goals, quantitative effect tracking, and operational control. The platform evolution follows a “split‑merge” principle, moving from a chaotic era of fragmented tools to a unified digital operations platform with nine modules (container‑based monitoring, workflow, log, DB management, automation, knowledge graph, etc.). Future “central” era aims to establish four centers: end‑to‑end observability, intelligent decision, operations data, and automated execution.

Future Outlook

Guangdong Mobile continues to explore new AIOps scenarios, including emerging technologies such as the metaverse and ChatGPT, seeking to integrate innovative concepts into operational practice.