How AI-Driven Event Intelligence Transforms Data Center Fault Management

Background

Rapid growth of virtualization and cloud computing has multiplied the scale of data‑center IT infrastructure, causing frequent hardware and software failures. Operators need an automated pipeline that can turn raw alerts into fault objects, analyse the root cause, and trigger remediation actions.

Overall Architecture

The Event Intelligent Analysis System (EIAS) implements a three‑stage fault‑handling pipeline: fault identification → fault analysis → fault disposition . It consumes normalized alerts from a unified event platform via Kafka, enriches them with CMDB attributes, and interacts with surrounding systems:

Automation platform : executes orchestration scripts (shell, Python) for remediation.

CMDB : provides object topology and relationship data.

ITSM : supplies change‑order and incident‑ticket information.

Big‑data platform : stores raw alerts, metrics, logs and supports data‑cleaning utilities.

Fault Identification

Alert Formatting

Incoming alerts are transformed into the EIAS internal schema. Missing fields (e.g., host IP, service name) are populated by querying the CMDB.

Fault Model Definition

A fault model describes how one or more alerts map to a logical fault instance. Each model contains:

Basic information : fault name, target object, fault type, description.

Rule set (any combination may be used):

Keyword rule : matches JSON fields such as summary or level using logical operators (AND, OR, NOT).

Time rule :

Immediate – create fault as soon as the alert arrives.

Fixed window – aggregate alerts that occur within a predefined interval after the first alert.

Sliding window – keep aggregating while new alerts arrive within the last n seconds.

Location rule : restrict aggregation to alerts originating from the same host, deployment unit, or physical subsystem.

Analysis decision tree reference that determines which analysis modules are invoked once the fault is created.

Fault Analysis

When a fault instance is generated, EIAS executes a customizable decision tree that can query multiple data sources:

Alert correlation : shows related alerts on the same object within the past 48 h.

Metric trends : fetches performance metrics for the affected object for the two hours preceding the fault.

Change records : retrieves ITSM change tickets linked to the object.

Log inspection : extracts relevant lines from application and system logs.

Link tracing : builds a transaction‑code‑centric topology to display upstream/downstream services.

Results are visualised as a topology tree where nodes with active alerts are highlighted.

The decision tree is authored by experts: each node defines a data‑source query and a boolean condition; leaf nodes are linked to disposition actions.

Fault Disposition

Disposition follows the model defined in the decision tree. It consists of two layers:

Orchestration : a sequence of scripts (shell or Python) that may include isolation, service restart, or circuit‑break steps. The orchestration engine ensures ordering and conditional branching.

Operation : the smallest executable unit, e.g., systemctl restart tomcat or a custom isolation command.

All actions are logged for post‑mortem analysis.

AI Empowerment

Automatic Fault Modeling

When no manual model exists, an AI module extracts salient keywords from the alert text, matches them against historical patterns, and creates a new fault model with default time (immediate) and location (same host) rules. The generated model is stored for future reuse and can be reviewed by an expert.

Automatic Fault Clustering

EIAS uses a BERT‑based encoder to compute cosine similarity between an alert’s summary and existing cluster centroids. The workflow is:

Optionally provide a human‑written fault description as an anchor.

Clean the alert text (remove noise characters, normalize case).

Encode the cleaned text with BERT and compute similarity to each cluster.

If the highest similarity exceeds a configurable threshold (e.g., 0.78), assign the alert to that cluster.

Otherwise create a new cluster with the current alert as its seed.

Update the cluster index and repeat for the next alert.

This unsupervised process runs in real time, covering 100 % of historical alerts and reducing manual rule maintenance.

Automatic Analysis Scheme Generation

Prompt engineering combines the current alert, related metric snapshots, and relevant emergency‑plan excerpts to form a query for a large language model (LLM) such as ChatGLM2 or Llama‑2. Because LLMs have token limits, emergency‑plan documents are pre‑indexed in a FAISS vector store. The steps are:

Retrieve the top‑k most similar emergency‑plan passages using FAISS.

Compose a prompt that includes the alert summary, metric summary, and the retrieved passages.

Send the prompt to the LLM and receive a textual analysis and remediation plan.

Emergency‑Plan Retrieval

During incident handling, the system can locate the most relevant procedural guidance by:

Exact string match on fault keywords.

Keyword extraction followed by inverted‑index lookup.

Semantic similarity search against the FAISS vector store.

The retrieved text is presented to the operator or fed back into the LLM prompt.

Key Integration Points

Kafka consumer reads the unified alert topic and acknowledges offsets after successful formatting.

CMDB API is called to enrich alerts with attributes such as host_ip, service_name, and topology links.

Automation platform API receives orchestration payloads in JSON, e.g.,

{
  "workflow_id": "wf_12345",
  "steps": [
    {"type": "script", "language": "shell", "content": "systemctl stop nginx"},
    {"type": "script", "language": "python", "content": "restart_service('nginx')"}
  ]
}

FAISS index is refreshed nightly with new emergency‑plan documents; incremental updates occur when a new plan is added.

Conclusion

The Event Intelligent Analysis System digitises operational expertise into fault models, leverages AI for automatic model creation, text‑based fault clustering, and LLM‑driven analysis, and executes remediation through an automation platform. By closing the loop from detection to self‑healing, the platform reduces mean time to repair (MTTR) and scales fault‑handling capabilities without proportional increases in manual effort.