Automate Fault Root‑Cause Detection in Massive IT Operations

Scale Effects and Cloud Increase Operations Complexity

In super‑scale internet companies, server fleets exceed hundreds of thousands and cloud migration diversifies workloads, making IT operations increasingly challenging. Traditional processes must continuously evolve.

The article introduces a network‑focused automated fault root‑cause localization technique to accelerate incident diagnosis and improve service availability.

Main Pain Points in Complex Operations

Proliferation of diverse monitoring platforms

Delayed communication between operation teams

Low sharing of alarm information

Inconsistent engineer expertise and low automation

Building an Operations Ecosystem Centered on Fault Localization

Unified fault entry with machine‑learning classification and inference to automatically generate cases and notify engineers.

Persist and analyze all data, feeding insights back to alarm and quality‑management systems to boost efficiency and risk management.

Brief Overview of Automated Fault Root‑Cause Localization

The system is a diagnostic expert system comprising a human‑machine interface, knowledge base, inference engine, interpreter, integrated database, and knowledge acquisition module. The knowledge base and inference engine are critical; the article focuses on binary decision‑tree rules.

System Architecture

Monitoring system – collects probe data and generates alerts.

Ingress system – aggregates and normalizes alerts.

Inference system – applies the expert decision tree to locate the root cause.

Notification system – disseminates the identified fault information.

Case Study: Network Fault Root‑Cause Localization

The fault inference algorithm uses a binary decision tree to consolidate alerts and intelligently pinpoint failures, reducing engineer investigation time.

Extract experience into a binary decision tree.

Segment alerts by time‑slice algorithm.

Feed grouped alerts into the decision tree for automatic reasoning.

Designing the Inference Tree

Alerts are hierarchical: router‑level (e.g.,

ROUTER_ID

,

CPU

,

TM

), board‑level, and port‑level (e.g.,

LINK‑NEW

). Each layer contains atomic and derived alerts. The principle is to report higher‑level, more fundamental alerts first, then move to lower‑level, derived ones.

Four Principles for Building the Inference Tree

Prioritize higher‑level alerts that are the root cause.

Prefer atomic alerts over derived ones.

Construct the tree based on observed alarm relationships.

Validate rules using expert knowledge and the knowledge base.

Three Implementation Approaches

Feature → inference engine → conclusion → validation → result (semi‑manual).

Self‑collected features → inference engine → conclusion → validation → result (simple ML).

Data → feature‑driven inference engine → conclusion → validation → result (intelligent ML).

Four‑Step Guide to Build Your Fault Root‑Cause System

Construct a CMDB with static (chassis, matrix, board, module, port) and dynamic (IP, routes, port status, traffic) data.

Standardize alarm formats for consistent feature extraction.

Map logical relationships between alarms (e.g., upstream/downstream dependencies).

Develop the inference tree with decision nodes and conditions derived from expert troubleshooting logic.

Following these steps yields an automated fault root‑cause localization system that continuously improves accuracy, boosts operational efficiency, and aligns IT operations with practices of leading internet companies.