Skip to content
Taodaji Cuts Troubleshooting 50% with Cloud-Native Observability Stack
Cloud Native 19 min read

Taodaji Cuts Troubleshooting 50% with Cloud-Native Observability Stack

Taodaji migrated from self-built SkyWalking and Elasticsearch to Alibaba Cloud's managed ARMS, SLS, Cloud Monitor 2.0, and STAROps, achieving 40% lower API latency pre-launch, minute-level fault localization, 50% faster troubleshooting, and 30% operational cost reduction through unified traces, logs, metrics, and AI-driven root cause analysis.

Alibaba Cloud Observability
Alibaba Cloud Observability
Alibaba Cloud Observability

Hubei Taodaji Supply Chain operates the B2B agricultural trading platform "Youdaji," where order concurrency is high and transaction chains are long. Any system stall or data inconsistency directly translates into delivery delays and inventory mismatches, making stability a business lifeline rather than a technical nice-to-have.

Background: The Build-vs-Buy Dilemma

As the new core system adopted a microservice architecture with deep call chains and high expected concurrency, the team initially planned to use a self-built SkyWalking + Elasticsearch (ES) stack. While workable at small scale, the exponential growth of Trace, Metric, and Log data exposed three structural problems that could not be solved by simply adding machines.

Three Core Challenges

Self-built cluster maintenance burden and stability risk: Continuous ES index optimization, shard/replica tuning, JVM tuning, and version upgrades consumed core R&D resources. Under peak loads, massive trace and log writes risked ES cluster instability — exactly when observability was needed most.

Data silos and slow end-to-end diagnosis: Traces, metrics, and logs lived in separate systems without a unified correlation key. During pre-launch stress testing, a slow core interface required engineers to manually jump across three UIs, three query syntaxes, and stitch timestamps and TraceIDs by hand. Average root-cause localization exceeded 30 minutes.

Alert noise and lack of sustainable capability: Static threshold alerts generated storms during traffic spikes or network jitter, burying the true root cause. The ad-hoc troubleshooting practices built for launch could not be retained as daily operational assets.

Solution: Four-Layer Cloud-Native Observability Platform

After comparing the self-built route with Alibaba Cloud's managed services, Taodaji chose ARMS + SLS + Cloud Monitor 2.0 + STAROps for two reasons: zero-maintenance managed services with smooth migration, and native integration of traces, metrics, logs, topology, and AI diagnosis.

(1) Managed Tracing & Smooth Migration: ARMS Replaces SkyWalking+ES

ARMS Application Monitoring ingests data via OpenTelemetry standard protocol and Java Agent bytecode instrumentation — zero business code changes. Existing SkyWalking instrumentation habits and trace views carried over seamlessly. ARMS supports asynchronous context propagation across thread pools and message queues, keeping TraceIds intact across complex calls. The managed storage and query engine eliminates ES index tuning, shard management, JVM tuning, and stability firefighting, guaranteeing second-level write and query stability under high concurrency.

(2) Unified View Breaking Data Silos: Cloud Monitor 2.0

Business logs are centralized into SLS Log Service with TraceIDs embedded as correlation keys. Cloud Monitor 2.0 unifies metrics, logs, and traces in a single platform, creating a closed observability data loop. Engineers can now jump from an abnormal span in the trace topology directly to the corresponding business logs in SLS at the exact timestamp, and vice versa. The previous 30-minute manual stitching process becomes a single TraceID-driven automatic correlation of "trace view + metric curve + log details."

(3) Dynamic Topology Visualization: UModel

UModel automatically builds a real-time full-chain topology — frontend → backend → middleware → database → containers — without manual diagramming. The topology refreshes automatically on deployments, dependency changes, or container scaling, turning a static outdated diagram into a live map with real-time health status. Clicking any node reveals its associated metrics, traces, and logs, providing a unified visual entry for capacity planning, bottleneck prediction, daily inspections, and blast-radius analysis.

(4) AI-Driven Diagnosis: STAROps

STAROps layers AI on top of Cloud Monitor 2.0's unified data (Metric, Log, Trace, Event). It performs automated root-cause reasoning by traversing the UModel topology backward along dependency edges, pinpointing the source node and propagation path amid alert storms. It also executes proactive inspections to detect risk precursors before they become incidents, shifting operations from reactive firefighting to proactive prevention. Alert noise reduction and topology-aware correlation let the team move from "manual trace digging" to "AI proposes root cause, human confirms."

Measurable Outcomes

Pre-Launch Optimization: 40% Latency Reduction

Using ARMS full-chain tracing down to method-level code lines and slow SQL statements, the team eliminated performance bottlenecks on core trading paths before launch. Core interface average response time dropped ~40%, ensuring smooth high-concurrency go-live.

Fault Localization: From >30 Minutes to Minutes

Unified view + TraceID-native linkage + STAROps topology-aware AI root cause analysis cut average fault localization time from over 30 minutes to minutes. Overall troubleshooting efficiency improved ~50%, MTTR drastically reduced. For a fresh-produce supply chain, faster diagnosis directly means fewer delivery delays and inventory mismatches.

Cost & Asset Sustainability: ~30% Ops Cost Reduction

Managed platform eliminated self-built cluster hardware, endless tuning, and dedicated firefighting headcount. Pre-launch performance baselines, alert rules, and capacity models solidified into daily operational capabilities. STAROps AI root cause analysis and proactive inspections transformed the observability stack from a one-off project expense into a reusable, growing long-term asset. IT infrastructure and personnel ops costs fell ~30%.

Future Roadmap

Full business-line coverage: Extend the same managed observability and intelligent diagnosis to all lines — fresh procurement, warehousing/logistics, processing, multi-level distribution — establishing a single operational standard and data dialect.

From post-mortem to pre-emption: Combine UModel dynamic topology, performance baselines, and STAROps proactive inspection with capacity forecasting and trend prediction to intervene before risks materialize (e.g., pre-scale services ahead of promotions).

From technical observability to business metrics: Align technical stability indicators with business continuity measures — order fulfillment timeliness, transaction success rate, inventory sync accuracy — so technical decisions carry a business-value yardstick.

Architecture diagram showing ARMS, SLS, Cloud Monitor 2.0, STAROps integration
Architecture diagram showing ARMS, SLS, Cloud Monitor 2.0, STAROps integration

Related Products: ARMS Application Monitoring (https://www.aliyun.com/product/arms), SLS Log Service (https://www.aliyun.com/product/sls), Cloud Monitor 2.0 (https://www.aliyun.com/product/cms), STAROps Intelligent Ops Assistant (https://www.aliyun.com/product/starops).

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

cloud-nativemicroservicesObservabilityOpenTelemetrySLSARMSSTAROpsCloud Monitor
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.