From Manual Troubleshooting to AI Automation: Building an Intelligent Diagnosis System for Lazada Ads

Background

System stability is critical for Lazada's advertising engine because outages directly affect ad revenue. The team faced three core problems: a flood of alerts from multiple monitoring platforms (GoldEye, Kmonitor, etc.) that are hard to distinguish, long manual investigation paths that rely on personal experience, and rapid business expansion (SS/SP to Miravia, Daraz, Smax) that outpaced diagnostic capabilities.

Goals

Diagnosis correctness > 70%

Smart‑diagnosis coverage ≥ 50%

False‑alarm rate < 20%

Solution Overview

A three‑layer Multi‑Agent system was built:

Noise‑Identification Layer : filters duplicate alerts, normal traffic spikes, and known benign errors.

Agent Diagnosis Layer : routes alerts to domain‑specific sub‑agents (SD chain, Smax chain, Error QPS).

Foundation Layer : knowledge base, MCP toolset, GoldEye integration, generic utilities, and a large‑model (Tongyi Qianwen) for natural‑language understanding and reasoning.

System Architecture

Noise Identification

The platform records the full alert chain in the Holo database and provides a multi‑dimensional MCP query interface. Four strategies are applied:

Duplicate‑alert detection by matching new alerts with historical records from the same country and product within the past 24 hours.

Business‑feature detection (e.g., post‑promotion traffic drop) using a maintained promotion calendar.

Cross‑metric correlation across products and countries within the past hour.

Continuous learning: human feedback on mis‑diagnoses is written back to Holo to refine rules.

These upgrades raised noise‑identification accuracy from 60% to 85%.

SD Chain Diagnosis

When a revenue‑drop alert arrives, the SD‑chain agent follows a five‑step process:

Extract structured fields (time, country, business line, metric) from the alert using LLM‑based NLP.

Locate the abnormal metric among traffic, exposure rate (PVR), click‑through rate (CTR), and cost‑per‑click (PPC) using real‑time data from the MCP tool.

Drill down to the root cause (e.g., traffic surge, exposure anomaly, PPC index issue).

Correlate with recent code releases, experiments, or other events.

Generate a structured report containing the diagnosis conclusion, abnormal metrics, and possible root causes.

Smax Chain Diagnosis

Smax business lines have low base volume and high volatility, so the agent combines three monitoring windows (real‑time, hourly‑aggregate, T+1) and applies a multi‑time‑scale strategy to avoid false positives.

Error QPS Diagnosis

The Error QPS agent classifies URP error‑QPS spikes into six categories, focusing on the three most common: single‑machine issues (~15%), traffic/level changes (~20%), and release‑induced problems (~60%). The workflow includes:

Fast exclusion of C1 and C2 using Kmon metrics and CPU water‑level trends.

Log aggregation and white‑list filtering to isolate new error messages.

Release correlation across platforms (SuezOps, Whaleshark, TPP) with confidence scoring.

White‑list filtering further reduces mis‑diagnoses.

Key Design Decisions

Multi‑LLM Staged Reasoning

Initially a single LLM performed the entire diagnosis, causing repeated reasoning and token waste. The process was split into two stages: the first LLM handles surface analysis (identifying the abnormal metric) and outputs a structured intermediate result; the second LLM receives this output as context and performs deep root‑cause analysis. This eliminates redundant inference and improves consistency.

MCP Tool Design

Instead of returning raw time‑series data, the MCP tool aggregates metrics (average, max) for the requested window and returns only the relevant indicator. This prevents context overflow and speeds up LLM inference.

Knowledge‑Base Organization

Each diagnostic scenario is stored as a Markdown document containing description, steps, tool calls, criteria, and real cases. Retrieval‑Augmented Generation (RAG) fetches the appropriate document based on alert content, enabling dynamic adaptation to new business domains.

Data‑Driven Operations & Continuous Optimization

Metrics tracked include diagnosis correctness, false‑alarm rate, diagnosis latency, human‑hand‑over time, and coverage (business domains, chain types, metric types). A feedback loop collects human confirmations, updates the knowledge base, and refines agent rules.

Case Study & Results

Coverage now includes Lazada SS/SP and Smax revenue chains; Miravia and Daraz are in progress. FY26 false‑alarm rate stayed below 20%. Diagnosis latency dropped from 10‑15 minutes (manual) to 1‑2 minutes (automatic), with an automatic closure rate above 70%.

Conclusion & Outlook

The Multi‑Agent intelligent diagnosis system has been production‑ready for over six months, delivering 5‑10× faster incident resolution and freeing engineers from 24/7 on‑call pressure. Future work will expand coverage to additional markets, solidify agent prompts as reusable skills, enhance continuous‑learning mechanisms, and introduce cross‑chain automated correlation.