How Lemon Retail Achieved 70% Alert Convergence and Minute-Level MTTR with AI-Driven Cloud-Native Observability

Background: Retail SaaS at Scale Demands Intelligent Operations

Zhejiang Lemon Information Technology provides integrated digital solutions for supermarkets, chain stores, and distributors across 12 verticals. Its product matrix covers POS, chain ERP, WMS, fund settlement, CRM, mini-programs, and AI smart scales. With over 20,000 stores in 200+ cities serving 1.5M+ brands, Lemon operates a multi-tenant SaaS platform where checkout and payment requests from hundreds of thousands of terminals peak simultaneously. Any latency spike directly causes in-store queues and transaction failures, making operations a frontline business concern rather than a backend support function.

Business Challenges

Fragmented Alert Governance and Storm Noise

POS SaaS, chain ERP, WMS, and payment accounting each relied on separate observability stacks — Log Service, Prometheus metrics, APM, and cloud product monitoring. Alert rules were configured and maintained in different consoles. As business expanded, rule count grew rapidly, leading to misconfigurations and gaps. During promotions and holidays, alert storms overwhelmed on-call engineers, burying real anomalies in noise.

Observability Data Hard to Produce and Consume

Internal R&D, customer success, and regional operations needed diverse dashboards: system-side API success rates, latency distributions, resource loads; business-side store online rates, checkout volumes, payment success rates, settlement timeliness. Building these required complex cross-data queries with long lead times and high error rates. Business users lacked query syntax skills, locking observability expertise in a few hands and slowing response to business rhythms.

Manual Pre-Checks and Post-Incident Investigation

Pre-incident, container clusters and cloud resources hosting multi-tenant workloads were inspected manually via console logins, lacking reusable inspection SOPs and systematic risk detection. Post-incident, a single checkout failure could originate from store network, POS client, SaaS application, database, or payment channel. Cross-system troubleshooting required switching platforms and manual correlation, heavily dependent on senior engineers' experience. Any delay in payment-path diagnosis directly impacted store experience and fund compliance.

Lemon needed not another monitoring tool, but a unified intelligent operations system that consolidates alert entry, lowers data consumption barriers, shifts defenses left, and lets AI complete cross-system localization.

Solution: Four-Layer Intelligent Operations on a Unified Data Foundation

Lemon selected Alibaba Cloud CloudMonitor 2.0 (CMS 2.0) and the STAROps full-domain intelligent operations platform as the core. Lemon had already accumulated logs, metrics, and traces in Alibaba Cloud's observability suite. STAROps uses UModel (Observability Model) to unify these into a full-chain digital twin, consumed by alert governance, auto-inspection, and root-cause capabilities. The platform requires no separate activation; it is accessible directly from the CloudMonitor 2.0 or SLS consoles, integrating smoothly with Lemon's existing observability stack.

Layer 1: Unified Alert Governance — Single Entry for Multi-System Alerts

Lemon adopted CloudMonitor 2.0 Alert Management Center to converge rules previously scattered across Log Service, Prometheus, APM, and cloud product monitoring into one platform. Engineers no longer jump between systems to configure rules. On top, STAROps' built-in Alert Rule Governance Skill automatically audits existing rules, identifying duplicates, unreasonable thresholds, and long-silent stale rules. Combined with alert-and-event analysis, peak-period alerts are converged, reducing alert volume by 70% and eliminating storm interference with on-call judgment.

Key mechanism: Rule governance skill automates audit → detects duplicate rules, bad thresholds, stale rules → alert-event analysis converges high-frequency alerts.

Layer 2: Natural Language Observability — One Sentence Generates a Dashboard

Leveraging STAROps' NL2SQL (Natural Language to SQL) capability, Lemon's R&D and operations staff describe observability needs in plain language; the platform auto-generates queries and dashboards. Typical use cases: query checkout API success rate trends for a region, compare payment-channel latency distributions during promotions, analyze settlement task delay distributions and failure details, aggregate alert handling and handover stats by business line. Dashboard production compressed from "request → schedule → write query" multi-step workflow to a single conversation, enabling both rapid output and direct consumption by business roles.

Result: Demand response cycle drastically shortened to minute-level.

Layer 3: Scheduled Intelligent Inspection — Repetitive Checks Handled by Digital Employees

Using STAROps' Mission (long-term task) mechanism, digital employees (Agents) execute inspections, analysis, and change operations on schedule or event-driven triggers, converting manual repetitive checks into reliable automated workflows. Deployed scenarios: container cluster health inspection, application health inspection, cloud service resource health inspection, payment and settlement data pipeline quality checks. Inspection results are stored as structured reports with historical diff comparison, surfacing resource bottlenecks and config drift before they become incidents. For high-risk actions like scaling and config changes, a built-in Human-in-the-Loop (HIL) mechanism requests manual confirmation, balancing automation with change safety. This layer upgrades pre-incident defense from manual spot-checks to automated guardianship.

Layer 4: Dedicated Ops Digital Employee — Cross-System Rapid Root Cause Convergence

Lemon customized a general-purpose digital employee into a dedicated SRE intelligent agent by combining default rules, Skills , and MCP tools , scoped to its system boundaries with defined responsibilities, permissions, and callable tools. When checkout or payment alerts fire, the digital employee triggers rapid root-cause analysis using the UModel-built operations digital twin that unifies applications, services, resources, topology, alerts, and changes. Assisted by AI analysis operators — metric anomaly detection, log clustering, trace analysis, and change replay — it performs causal reasoning across system boundaries. This transforms the previously expert-dependent, multi-console troubleshooting process into reusable automated analysis, directly addressing the long fault-localization chain challenge.

Realized Value: Three Upgrades from Reactive Firefighting to Proactive Autonomy

Alert Governance: From Scattered Manual Work to Unified Standard Actions

Before: Four observability systems had separate alert entry points; rule configuration relied on personal experience; misconfigurations and gaps went unnoticed; peak storms flooded screens; engineers hunted real anomalies in noise. After: Unified alert center plus rule governance skill turns rule creation, review, and optimization from experience-dependent manual work into tool-supported standard actions. Peak alerts converged on demand; on-call judgment no longer disrupted by storms. Alert convergence reached 70%.

Data Access: From Request-and-Wait to One-Conversation Dashboards

Before: A business dashboard required request, scheduling, complex query writing — long cycle, error-prone; non-technical staff waited for experts. After: With NL2SQL and Q&A data retrieval, R&D, customer success, and operations personnel fetch core application metrics and store-level business data in natural language without query syntax. Observability capability spreads from few experts to broader business roles. Demand response cycle cut to minute-level.

Operations Posture: From Post-Incident Response to Pre-Incident Protection

Before: Resource inspections manual, coverage and consistency unreliable; fault localization via cross-platform switching and manual correlation; MTTR hard to compress. After: Under a two-week release cadence, Mission scheduled inspections continuously produce structured reports with historical diffs, catching config drift early; automated inspection provides stable quality gate for releases. On failure, dedicated digital employee triggers rapid root-cause analysis from alert or Q&A, leveraging digital twin topology and AI operators to narrow scope across systems, cutting engineer time spent switching consoles and manually correlating. Checkout and payment path MTTR steadily compressed to minute-level.

Future Outlook: Expanding the Boundaries of Proactive Autonomy

From Core-Path Pilot to Full Business-Line Coverage

Extend digital employee responsibilities from checkout and payment paths to chain ERP, WMS, mini-program new retail, and more systems. Reuse validated alert governance, inspection, and root-cause capabilities as standard configuration across all product lines, making intelligent operations a default capability.

From Technical Metrics to Business Continuity Measurement

Using NL2SQL cross-data-source query ability, connect technical indicators (API success rate, latency) with business indicators (store online rate, checkout volume, settlement timeliness) to build a business-continuity measurement system, letting operations value be measured directly in business language.

From Human-Machine Collaboration to Higher Autonomous Operations

Within the HIL safety framework, gradually expand digital employee autonomous closed-loop scenarios. Continuously convert accumulated experience from inspection, localization, and changes into reusable skill assets, evolving operations expertise from individual capability into organizational asset.