Lemon's Intelligent Ops: 70% Alert Convergence, Minute-Level MTTR for 20K+ Retail Stores

Background: High-Stakes Checkout Peaks

Lemon (Zhejiang Lemon Information Technology) provides integrated retail digitalization solutions covering POS, ERP, WMS, payment settlement, CRM, and AI-powered devices for 20,000+ stores across 200+ cities. During business peaks, hundreds of thousands of store terminals simultaneously generate checkout and payment requests; any latency spike directly causes queueing and transaction failures. Lemon must withstand massive peak loads while maintaining multi-tenant SaaS stability under bi-weekly release cycles.

Business Challenges

Fragmented Alert Governance and Alert Storms

POS SaaS, chain ERP, WMS, and payment accounting systems each relied on separate observability stacks — Log Service, Prometheus metrics, APM, and cloud product monitoring. Alert rules were configured and maintained in different consoles, leading to misconfigurations and omissions. During promotions and holidays, alert storms overwhelmed on-call engineers, burying real anomalies in noise.

Observability Data Hard to Produce and Consume

Internal R&D, customer success, and regional operations needed diverse dashboards: system-side API success rates, latency distributions, resource loads; business-side store online rates, checkout volumes, payment success rates, settlement timeliness. Building these required complex cross-data-source queries with long lead times and high error rates. Business users lacked query skills, locking observability expertise in a few hands and slowing response to business rhythms.

Manual Pre-Checks and Post-Incident Investigations

Pre-incident, multi-tenant container clusters and cloud resources were inspected manually without reusable inspection SOPs, missing resource bottlenecks and latent risks. Post-incident, a single checkout failure could originate from store network, POS client, SaaS application, database, or payment channel. Cross-system troubleshooting required switching across platforms and manual correlation, heavily dependent on senior engineers' experience. Any delay in payment-chain diagnosis directly impacted store experience and fund compliance.

Lemon needed not another monitoring tool, but an intelligent Ops system that unifies alert entry, lowers data-consumption barriers, shifts defenses left, and lets AI perform cross-system root cause analysis.

Solution: Four-Layer Intelligent Ops on a Unified Data Foundation

After evaluation, Lemon adopted Alibaba Cloud CloudMonitor 2.0 (CMS 2.0) and the STAROps full-domain intelligent Ops platform as the core. Lemon had already accumulated logs, metrics, and traces in Alibaba Cloud's observability suite. STAROps uses UModel (Observability Model) to unify these into an end-to-end digital twin, which is then consumed by alert governance, auto-inspection, and root-cause analysis capabilities. The platform requires no separate activation; it is accessible directly from the CloudMonitor 2.0 or Log Service (SLS) consoles, integrating smoothly with Lemon's existing observability stack.

(1) Unified Alert Governance: Converging Multi-System Alerts into One Entry

Lemon uses CloudMonitor 2.0 Alert Management Center to consolidate alert rules previously scattered across Log Service, Prometheus, APM, and cloud product monitoring into a single platform for unified management, eliminating multi-console jumps. On top of this, STAROps' built-in Alert Rule Governance Skill automatically audits existing rules, identifying duplicate rules, unreasonable thresholds, and long-dormant ineffective rules. Combined with alert and event analysis, high-peak alerts are converged, significantly reducing alert-storm erosion of on-call efficiency.

(2) Natural Language Observability: One-Sentence Dashboard Generation

Leveraging STAROps' NL2SQL (natural language to query) capability, Lemon's R&D and operations staff can describe observability needs in plain language; the platform auto-generates corresponding queries and dashboards. Typical uses: querying checkout API success-rate trends for a region, comparing payment-channel latency distributions during promotions, analyzing settlement-task delay distributions and failure details, aggregating alert handling and handover stats by business line. Dashboard production is compressed from a multi-step "request → schedule → write query" process into a single conversation, enabling rapid output and direct consumption by business roles.

(3) Scheduled Intelligent Inspection: Handing Repetitive Checks to Digital Employees

Lemon uses STAROps' Mission (long-term task) mechanism to let Agents (digital employees) execute inspections, analysis, and change operations on a scheduled or event-driven basis, converting previously manual repetitive checks into reliable automated workflows. Inspection scenarios include container cluster health, application health, cloud-service resource health, and payment/settlement data-pipeline quality checks. Results are stored as structured reports with historical diff comparisons, enabling early detection of resource bottlenecks and configuration drift before they become failures. For high-risk actions like scaling and config changes, Mission's built-in HIL (Human-in-the-Loop) mechanism triggers manual confirmation requests, raising automation levels while guarding the change-safety baseline. This layer upgrades the pre-incident defense from manual item-by-item review to automated guardianship.

(4) Dedicated Ops Digital Employee: Rapid Cross-System Root Cause Convergence

Lemon combines default rules, Skills , and MCP tools to customize a general-purpose digital employee into a dedicated SRE agent tailored to its business, configuring its responsibilities, permissions, and callable tools according to system boundaries. When a checkout or payment alert fires, the digital employee can trigger rapid root cause analysis based on alert context or an engineer's natural-language question. Using the UModel -built Ops digital twin, it correlates applications, services, resources, topology, alerts, and changes, and applies AI analysis operators — metric anomaly detection, log clustering, trace analysis, and change replay — to perform causal reasoning across system boundaries. This transforms the previously experience-dependent troubleshooting process into reusable automated analysis, directly addressing the long fault-localization chain challenge.

Realized Value: Three Upgrades from Reactive Firefighting to Proactive Autonomy

(1) Alert Governance: From Fragmented Manual Work to Unified Standard Actions

Previously, four observability systems had separate alert entry points; rule configuration relied on personal experience, making misconfigurations hard to detect. Peak periods flooded screens with alerts, forcing engineers to hunt for real anomalies in noise. Now, the unified alert management center plus rule governance skills turn rule creation, review, and optimization from experience-dependent manual work into tool-supported standard actions. Peak alerts are converged on demand, and on-call judgment is no longer disrupted by storms. Alert convergence reached 70%.

(2) Data Access: From Request-and-Wait to One-Conversation Dashboards

Previously, a business observability dashboard required request submission, scheduling, and complex query writing — long cycles, error-prone, and inaccessible to non-technical staff. Now, with NL2SQL and Q&A data retrieval, R&D, customer success, and operations personnel can obtain core application observability metrics and store-operation data in natural language without query syntax. Observability capability spreads from a few experts to broader business roles, cutting demand-response cycle to minutes.

(3) Ops Posture: From Post-Incident Response to Pre-Incident Protection, with AI-Accelerated Troubleshooting

Previously, resource inspections were manual spot-checks with inconsistent coverage; fault diagnosis relied on cross-platform switching and manual correlation, making MTTR hard to compress. Now, under a two-week release cadence, Mission scheduled inspections continuously produce structured reports and historical diffs, identifying configuration drift early and providing a stable safety net for release quality. When incidents occur, the dedicated digital employee triggers rapid root cause analysis directly from alerts or Q&A, leveraging the digital twin topology and AI analysis operators to narrow the investigation scope across systems, reducing engineer time spent switching consoles and manually correlating data. Average recovery time for checkout and payment chains is steadily compressed to minutes.

Future Outlook: Expanding the Boundaries of Proactive Autonomy

The four capabilities — unified alert governance, natural language observability, automated inspection, and intelligent root cause analysis — are now running on core paths. Lemon will deepen collaboration with Alibaba Cloud in three directions:

(1) From "Core-Path Pilot" to "Full Business-Line Coverage"

Extend digital employee responsibilities from checkout and payment chains to chain ERP, WMS, mini-program new retail, and more systems, reusing validated alert governance, inspection, and root-cause capabilities as standard configurations for every product line, making intelligent Ops a default across all business lines.

(2) From "Technical Metric Observability" to "Business Continuity Measurement"

Using NL2SQL's cross-data-source query ability, further connect technical metrics (API success rate, latency) with business metrics (store online rate, checkout volume, settlement timeliness) to establish a business-continuity measurement system, making Ops value directly measurable in business language.

(3) From "Human-Machine Collaboration" to "Higher-Degree Autonomous Ops"

Within the safety framework of human-in-the-loop, gradually expand the scope of digital employee autonomous closed-loop scenarios, and continuously convert accumulated experience from inspection, diagnosis, and change into reusable skill assets, evolving Ops expertise from individual capability into organizational assets.