This article details how CloudLens for Elasticsearch leverages Cloud Monitoring 2.0 and UModel to unify metrics and logs per instance, then applies STAROps' four-round diagnostic framework—scoping, drilling down, log verification, and conditional recommendations—to transform ad-hoc troubleshooting into evidence-based, reproducible investigations.
This comprehensive guide covers DCGM Exporter deployment modes, metric semantics, sampling internals, version-specific breaking changes, PromQL pitfalls, and three real-world failure cases with root-cause analysis and alerting rules for production GPU monitoring.
This article explores why traditional capacity planning fails in microservices architectures and presents automated performance testing as a neural center for dynamic planning, detailing a PDCA-Perf closed-loop model with self-generated scenarios, adaptive load, and correlated metrics, while warning against tool-centric and static approaches.
Based on auditing 27 AI-native applications, this article identifies five common prompt testing pitfalls: treating testing as casual Q&A, ignoring multi-turn context, using accuracy as a universal metric, lacking production-like sandbox environments, and failing to version test assets, with concrete examples and improvement strategies for each.
Lenovo Baiying partnered with Alibaba Cloud AI Gateway to create Token Plan and Token Key, enabling unified model access, fine-grained metering, traffic scheduling, and FinOps for scalable AI service delivery.
Infinitus built a seven-stage data flywheel using Alibaba Cloud AgentLoop to continuously improve their Lingzhi AI agent, automating trace collection, evaluation, annotation, and experience mining while adding human review gates to turn production traces into reusable prompt optimizations, failure patterns, tool paradigms, and business SOPs.
This comprehensive guide covers distributed tracing implementation with SkyWalking and Jaeger, detailing trace data costs, head vs tail sampling trade-offs, architecture differences, capacity planning formulas, probe integration pitfalls, cross-language trace breakage causes, and three real-world troubleshooting cases with concrete metrics and configurations.
This article details building production-grade GPU cluster observability using DCGM Exporter and Prometheus, covering hardware counter metrics (SM_ACTIVE, DRAM_ACTIVE), Kubernetes pod mapping, MIG monitoring, alerting rules, and real-world troubleshooting cases like NVLink degradation and thermal throttling.
The article explains how LoongSuite Pilot normalizes telemetry from 20+ AI coding agents into a unified GenAI event schema with three-level session structure, token breakdowns, cost fields, and provider normalization, while its Qoder plugin achieves near-100% collection coverage via silent installation, self-healing, auto-updates, and parameter-fingerprint-based idempotent upgrades — all driven by a single admin configuration pushed through Qoder's enterprise marketplace.
AI codingAgent NormalizationAutomated Distribution
The team transformed an Android chat assistant with zero tests and CI into a robust product through 21 PRs, establishing a five-layer safety net, enforcing privacy-first design, optimizing performance via trace-driven analysis, hardening reliability with black-box testing, and adding on-device observability—all while uncovering flaws in their own test infrastructure.
This article explains why DeepSeek Harness requires managed hosting for enterprise use, detailing four deployment scenarios, three integration modes, identity and network governance, observability layers, a seven-step rollout sequence, and deterministic evaluation methods to turn a solo agent tool into shareable, auditable infrastructure.
This article compares Thanos and VictoriaMetrics for Prometheus high availability, covering architecture differences, deployment steps, configuration details, compression ratios, query capabilities, cost models, migration paths, troubleshooting, and operational best practices to help teams choose the right solution based on scale, retention needs, and team capacity.
Netdata is an open-source, distributed real-time monitoring solution that collects per-second metrics, auto-generates dashboards, runs ML anomaly detection, and provides AI-driven analysis with zero configuration, supporting Linux, Docker, Kubernetes, and integrations with Prometheus, OpenTelemetry, and AI assistants via MCP.
A comprehensive guide to building a three-layer observability stack for vLLM inference services, covering GPU metrics (DCGM), engine-level queue and KV cache metrics, request-level latency (TTFT/TPOT), Prometheus alerting rules, Grafana dashboard design, and troubleshooting runbooks with real-world case studies.
A comprehensive guide to deploying Loki with Promtail/Alloy for centralized logging, covering architecture decisions, label design, single-binary deployment with object storage, LogQL query patterns, monitoring, troubleshooting, and migration from Promtail to Alloy.
This article explains why AI Gateways are increasingly adopted, detailing their core functions—protocol normalization, model routing, key management, and observability—with a practical LiteLLM implementation guide, plus pros, cons, and ideal use cases for multi-model, cost-sensitive, and enterprise environments.
Alibaba's Qoder team details their production vector retrieval architecture for AI coding agents using Alibaba Cloud Elasticsearch Agent Engine, mapping each codebase to a dedicated slice for multi-tenant isolation, achieving sub-100ms P95 latency, hundred-billion document scale, and cost-efficient tiered storage across 30+ write and 5+ query ECS instances.
This article analyzes when to adopt Service Mesh over SDK-based governance, using three real incidents to illustrate trade-offs, comparing three governance generations, detailing cost analysis, migration paths, and troubleshooting methods, concluding that Mesh amplifies existing governance capability rather than creating it.
A comprehensive guide to consolidating multi-layer Nginx into a single programmable OpenResty gateway, covering Lua request phases, shared-memory rate limiting algorithms (leaky bucket, token bucket, distributed), dynamic configuration, authentication ordering, risk control tiers, performance tuning, and six production troubleshooting scenarios with concrete fixes.
This article provides a comprehensive decision framework for choosing between Layer 4 and Layer 7 load balancing, covering protocol analysis, TLS termination, health checks, session persistence, canary deployments, observability, and rollback procedures using HAProxy as a reference implementation.
This article details the architecture and implementation of production-grade AI agents using Alibaba Cloud's AgentCore and the Codex harness, covering core components like models, tools, skills, storage, identity, observability, and multimodal support, plus a step-by-step deployment tutorial with environment configuration and demo results.
This article details the principles of Agent Harness self-evolution, covering multi-agent architecture, four-layer observability, recursive self-verification, and compares three technical approaches—Google's Harness², Fudan's AHE, and EverMind's HarnessBank—with engineering implementation steps for building self-improving AI agent systems.
Yidian Tianxia built a multi-agent IT operations system on Alibaba Cloud AgentCore, achieving 68-87% efficiency gains across four scenarios and reducing root-cause analysis from 40 to 8 minutes by replacing ad-hoc AI chat with a Leader-Worker agent architecture, unified data layer, quality gates, and continuous self-evolution loops.
The article details how the Firefly AIGC platform evolved from a chat interface to a production-grade content generation system, covering its unified orchestration architecture, differentiated execution using Ray Serve and Ray Data, multimodal data lakehouse with Lance, and workflow observability via Trace.
This article details the architecture of the Dayu AI Model Gateway, covering its three-plane design (control, data, accounting), streaming protocol conversion across OpenAI, Anthropic, and Gemini formats, fault-tolerant scheduling with retry budgets, granular observability metrics, immutable price snapshots for accurate billing, and resource quota management with distributed consistency guarantees.
This article analyzes the Agentic framework's architecture, showing how its event-driven design, Ray-based runtime, and typed event streams solve production challenges like hallucination, debugging, and human-in-the-loop that first-generation agent frameworks ignore.
This article explains the distinct roles of LangChain, LangGraph, and LangSmith in the LangChain ecosystem: LangChain provides standardized components for linear LLM workflows, LangGraph enables stateful, branching agent orchestration with loops and human-in-the-loop, and LangSmith offers observability through tracing, evaluation, and prompt optimization for production quality.
This article establishes a four-tier alert classification framework and provides concrete PromQL queries for user-facing errors, latency percentiles, saturation signals, and resource risks, showing how to route only actionable alerts via Alertmanager while demoting noisy host metrics to dashboards.
Taodaji migrated from self-built SkyWalking and Elasticsearch to Alibaba Cloud's managed ARMS, SLS, Cloud Monitor 2.0, and STAROps, achieving 40% lower API latency pre-launch, minute-level fault localization, 50% faster troubleshooting, and 30% operational cost reduction through unified traces, logs, metrics, and AI-driven root cause analysis.
The article argues that the Harness — the agent runtime framework — is the true product for production AI agents, detailing 12 architectural principles covering task-oriented tool discovery, layered memory, credential isolation, approval workflows, code execution sandboxes, multi-channel adapters, state persistence, and system evaluation.
The Rework 2026 report reveals only 20% of enterprises have mature AI Agent governance frameworks, leaving 80% exposed to risks like unregistered agents, excessive permissions, poor observability, and compliance gaps; the article outlines core governance elements, technical architecture using MCP/A2A/Cedar/AOP, lifecycle management processes, and a pragmatic three-step adoption path.
This article evaluates eight AI-powered and foundational operations tools for 2026 across alerting, logging, metrics, root cause analysis, and self-healing, providing applicable scenarios, limitations, and a team-size-based selection guide while emphasizing that solid observability foundations must precede AI adoption.
JD's Firefly platform shares its engineering practices for scaling AIGC from single model calls to a production-grade content generation system, covering unified orchestration with DAGs, differentiated execution via Ray Serve and Ray Data, multimodal data lakehouse with Lance, and workflow-level observability through Trace.
JD Retail's Firefly AIGC platform moves beyond single-model generation to scalable content production by unifying orchestration, differentiated execution via Ray Serve and Ray Data, a Lance-based multimodal data lakehouse for material governance, and end-to-end traceability across DAG workflows.
This article outlines eleven common misconceptions in Harness Engineering—from treating it as advanced prompting to neglecting observability and memory management—that cause AI agent teams to fail, emphasizing that Harness must be iteratively built from real failure modes, not copied, and that human engineering discipline is the true foundation.
This article dissects LangGraph's architecture using a ticket pre-audit trace, explaining how its graph definition, state management, Pregel-based runtime, persistence checkpoints, and observability integrate to enable reliable, recoverable AI agent execution in production systems.
Alibaba's Qoder team details how they built a production-grade vector retrieval system for Agentic Coding using Alibaba Cloud Elasticsearch, handling 100B+ documents with per-codebase slices for multi-tenant isolation, achieving 14ms P95 latency, 2GB/s write throughput, and ~0.016 yuan monthly cost per codebase, while sharing indexing, query pipeline, observability, and future unified retrieval plans.
Alibaba Cloud details its methodology for constructing and maintaining high-quality evaluation datasets for the STAROps AIOps agent, covering fault injection, signal capture, evidence extraction, quality checks, environment construction, and continuous iteration to ensure reliable root cause analysis assessments.
Claude Code's tool_decision events only capture accept/reject with a generic source field merging eight judgment sources, while the newer safety_event adds 14 categories but suffers from join gaps, attribute truncation, and missing events for unattended rejections, requiring joined audit tables and CI validation to identify actual blockers.
Alibaba Cloud SLS Session Store transforms OpenTelemetry traces and logs into standardized execution trajectories, enabling continuous data accumulation for Agent observability, dataset construction, LLM-based evaluation, and recursive self-improvement (RSI) loops, as demonstrated through a coding agent debugging case study.
Agent Self-EvolutionAlibaba CloudDataset Construction
Traditional Prometheus monitoring missed micro-bursts, TCP zero-windows, fsync storms, conntrack exhaustion, and startup file traversal — eBPF kernel-level tracing revealed all five root causes in a production Kubernetes cluster.
This article details AgentCore, an enterprise-grade agent platform, covering its ten core capabilities: controlled execution with runtime/sandbox/tool boundaries, full-chain observability with trace metrics, context engineering unifying memory and knowledge, multi-agent team collaboration with three lifecycle modes, evaluation via Code/LLM/Agent judges, skill lifecycle management with six-stage governance, three-layer asset catalog, identity/security with triple identity and audit trails, cost FinOps with per-call metering and quotas, and full lifecycle management from build to continuous improvement.
Yidian Tianxia shares how they built a unified five-layer AI middle platform — model, capability, agent, collaboration, and observability — enabling 100+ daily active agents, 130+ MCPs, and 1,500+ skills to autonomously execute end-to-end ad campaigns while keeping humans in the loop for high-risk decisions, achieving 40% ops efficiency gains and targeting 80% task closure.
This article explains why AI agents become slower, costlier, and less stable when deployed to real users, detailing the five-layer cost stack (tokens, tools, retries, context growth, human ops) and providing actionable strategies for budgeting, observability, and cost control.
This article systematically categorizes and explains core microservice design patterns across decomposition, integration, database, event-driven, observability, and cross-cutting concerns, providing practical guidance for architects building resilient, scalable distributed systems.
This article details how to design multi-agent systems for production incident handling, covering responsibility division by decision authority and toolsets, structured handoff contracts, state management, permission isolation via MCP gateways, LangGraph orchestration with human-in-the-loop approvals, observability, and cost control.
This article analyzes 11 critical misconceptions teams encounter when implementing Harness Engineering — the practice of constraining AI agents within structured runtime environments — covering cognitive errors like confusing Harness with advanced prompting, practical gaps such as missing observability and flawed self-evaluation, and organizational failures including neglecting human discipline and enterprise constraints, backed by benchmark data from OpenAI, LangChain, Anthropic, and MiniMax.
This article presents a production-grade observability design for agent systems, covering task modeling, trace propagation, lease/fencing/checkpoint mechanisms, OpenTelemetry instrumentation in Go and Java, MCP integration, Kubernetes collector setup, PromQL dashboards, Argo CD release correlation, and fault injection experiments for failure attribution.
This article details Alibaba Cloud ACK's fault self-healing system for heterogeneous compute clusters, covering its Kubernetes-native architecture, multi-layer detection and event conversion, orchestrated repair workflows with stepwise authorization, rate limiting and circuit breaking, full observability, and practical deployment strategies to maximize effective compute throughput.
Netdata is an open-source monitoring agent that auto-discovers 800+ integrations, collects metrics every second, runs 18 unsupervised ML models locally for anomaly detection, and uses ~5% CPU/200MB RAM at 5000 metrics/sec — all without configuration.
This guide presents a systematic, evidence-based approach to diagnosing Linux CPU spikes using six core commands and deeper analysis of processes, threads, cgroups, and runtime profiling to distinguish between user-space computation, system calls, I/O wait, soft interrupts, and container throttling.
This comprehensive guide details building a unified Prometheus monitoring stack for enterprises, covering host metrics with node_exporter, container metrics via cAdvisor and kube-state-metrics, business instrumentation, service discovery, alerting with Alertmanager, Grafana dashboards, capacity planning, cardinality management, high availability, and troubleshooting real-world incidents.
This article explains how Alibaba Cloud's AgentLoop platform continuously audits Dify AI applications by tracing model and tool calls, detecting sensitive data exposure, indirect prompt injection, and privilege misuse, and enabling custom rules to pinpoint root causes in workflow nodes for effective remediation.
Alibaba Cloud details its open-source stack for AgenticOps — LoongSuite/OBI for standardized OpenTelemetry collection, UModel for semantic fault context, and BRiSE for reproducible benchmarking — enabling reliable production-grade AI operations.
This article outlines a structured approach to effectively using coding agents, emphasizing architecture-first decisions, specification and context engineering over prompt engineering, a five-element task delegation model (Goal, Scope, Context, Constraints, Done), the shift from assistant to autonomous agent, and rigorous verification, testing, and observability as essential guardrails.
This article explains distributed tracing concepts, the OpenTracing standard, and how SkyWalking implements automatic span collection, context propagation, unique trace IDs via Snowflake, and adaptive sampling to minimize performance overhead.
This final article in the AI Agent development series details five pillars for production-ready agents—evaluation systems, observability, safety guardrails, cost/latency optimization, and deployment/operations—illustrated with real incidents, code examples, and a 30-item checklist.
Explores observability fundamentals including its three pillars (logs, metrics, tracing), structured logging with Serilog message templates, OpenTelemetry distributed tracing with W3C TraceContext propagation, ASP.NET Core health checks for Kubernetes liveness/readiness probes, and production monitoring architecture with correlation, standardization, sampling, and alerting strategies.
This article compares five leading AI gateways—Bifrost, Cloudflare AI Gateway, Vercel AI Gateway, LiteLLM, and OpenRouter—evaluating their performance benchmarks, model routing capabilities, deployment options, observability, cost control, and suitability for different production environments.
Alibaba Cloud details its open-source AgenticOps foundation comprising LoongSuite/OBI for standardized OpenTelemetry data collection, UModel for unified semantic fault context modeling, and BRiSE for reproducible benchmark evaluation, enabling reliable AI-driven operations from demo to production.
This article details a high-throughput WeChat Mini Program login architecture in Go targeting 50k entry QPS, covering distributed session design, async peak shaving with Kafka, capacity modeling using Little's Law, API contracts for sync/async flows, idempotency, transactional outbox, token management, observability, and load testing strategies.
The article explains how successful Agent execution traces can still produce wrong answers, using a procurement case where an Agent cited an outdated contract attachment. It details a systematic approach: trace errors through observability across indexing, retrieval, context assembly, and generation; convert feedback into reproducible test samples with versioned inputs and expected behaviors; evaluate using rule-based checks, model-assisted judging, and human review; separate critical error categories instead of relying on a single score; maintain regression suites with both failure and success cases; and rerun tests in isolated environments while recording versions to distinguish new results from original incidents.
This chapter expands monitoring beyond server resources to business results, covering server, business, log monitoring, active reporting, and passive scanning to detect silent failures, and outlines a seven-step design process with alert deduplication, escalation, and reconciliation scans.
This article explains why SRE agents can reach wrong root causes even with authentic evidence, using a case study where a pre-deployment log from v41 was incorrectly linked to v42 latency spikes. It details methods to align timestamps, entity versions, and metric windows, then apply counter-evidence queries and offline replay validation to improve diagnostic accuracy.
This article compares five leading LLM testing tools across evaluation dimensions, CI/CD integration, interpretability, and debugging capabilities, using real-world case studies to highlight strengths, limitations, and selection guidance for production-grade model quality assurance.
This practical guide compares open-source load testing tools (JMeter, Locust, k6, Artillery), details script development with realistic user behavior and data injection, explains environment isolation via traffic labeling and shadow databases, and teaches multi-dimensional result analysis beyond basic TPS metrics.
Alibaba Cloud SLS provides 8 Agent Skills covering the full log lifecycle — collection, indexing, query, alerting, visualization, and root cause analysis — enabling natural language-driven log management with concrete installation commands and practical scenarios for monitoring and troubleshooting.
Huajiao Technology details taking an AI-powered guessing game from demo to production in their live streaming app, covering mechanics, AI-assisted development, two major reworks (legacy code modification and position calculation bug), observability improvements, and post-launch metrics showing 7.9 daily rounds per room with 77% from repeat plays.
AI engineeringAI-assisted developmentObservability
This article contrasts traditional resource-driven response time optimization with modern path-driven paradigms, exposing three common traps and introducing user-journey metrics, latency budgets, and proactive shaping via eBPF to deliver predictable user experiences in distributed systems.
This article explains how to design observable Agent task statuses by distinguishing tasks, actions, and sessions; tracking execution phases, wait reasons, and action results; separating activity from real progress; and building task observation cards with evidence links for debugging and human handoff.
This guide covers end-to-end OpenTelemetry implementation for unified observability, including zero-code auto-instrumentation via Kubernetes Operator, Collector deployment patterns with memory limiting and batch processing, head/tail sampling strategies for cost control, migration from Jaeger/Prometheus, log collection with trace correlation, and six critical production pitfalls to avoid.
Arthas 4 introduces an MCP server that lets AI assistants execute Java diagnostic commands like thread, trace, and watch through natural language, enabling developers to troubleshoot JVM issues without memorizing command syntax, with 29 tools across JVM, class loading, monitoring, and utility categories.
Alibaba Cloud's AgentCore product lead details the platform's architecture for building, governing, and scaling enterprise AI agents, covering collaboration patterns, asset management, observability, evaluation methods, and security governance.
An interviewer shares insights from three Agent developer candidates who had impressive resumes but struggled with production concerns like tool error handling, RAG evaluation, memory design, loop control, and observability, highlighting the gap between demo-building and production-ready systems.
This article explains why traditional monitoring fails for AI agents and introduces AgentOps—covering observability with full traces, five key SLOs (task success rate, human escalation rate, cost per successful task), a multi-stage release pipeline with shadow and canary phases, FinOps with model routing, ROI calculation including risk costs, agent lifecycle management, and a continuous improvement flywheel turning production traces into eval datasets.
This chapter summary traces the evolution from monolith to microservices, detailing core components like service discovery, circuit breakers, gateways, and observability, while stressing that adoption must be driven by business complexity and team needs — not service count — and provides a practical checklist for splitting and transformation.
WeChat's technical architecture team shares their exploration of adapting OLAP infrastructure for Agentic AI, covering distributed observability with Langfuse and ClickHouse, remote memory services using vector search and FUSE, and access protection mechanisms treating agents as authenticated identities.
The article analyzes risks of AI-generated 'vibe coding' where engineers lose internal code understanding, detailing consequences like hidden architectural decay, loss of tacit production knowledge, and inability to debug complex failures, then proposes practical strategies: defining white-box boundaries, independent verification, limiting change scope, retaining takeover capability, preserving dark knowledge, and reallocating engineering time toward constraint design and failure recovery.
AI-assisted developmentObservabilityblack-box systems
This guide details building an enterprise-grade observability system using SkyWalking, covering distributed tracing, metrics monitoring, log correlation, and alert governance to eliminate blind troubleshooting in microservices.
A hands-on test of YuanNao Web Agent reveals five key design patterns — unified multi-agent portal, model-agent decoupling, transparent metering, enterprise usage dashboards, and layered security — that transform personal AI agents into a centrally managed, cost-trackable, and secure enterprise asset.
This article details a production-grade architecture for handling high concurrency in AI agent systems, covering real load estimation, admission control with bounded queues, async task processing, hierarchical concurrency budgets for models and tools, resource isolation via bulkheads, idempotency for state consistency, graded degradation strategies, and observability-driven capacity planning.
This article traces software architecture evolution from monoliths through primitive distributed systems, SOA, microservices, and cloud-native, highlighting two recurring patterns — finer decoupling and stronger fault isolation — and a shift from zero-failure goals to designing for resilience, while outlining four unprecedented challenges AI agents introduce: semantic hallucinations, stateful context, dynamic orchestration, and observability gaps.
This article clarifies the distinct roles of LangChain (component library), LangGraph (stateful workflow orchestration), and LangSmith (observability & evaluation) in the LangChain ecosystem, explains how they collaborate, and provides a selection guide for different AI application scenarios.
A Go proposal (#79179) provisionally accepted to add a memory_space sample type to heap/alloc profiles, providing a full memory hierarchy from RSS down to live/dead heap, stacks, runtime, and non-Go memory, solving the long-standing gap where heap profiles show only about 50% of actual RSS.
This article explains that observability is not centralized logging but a practice of defining business outcomes (SLIs/SLOs), correlating metrics, logs, and traces via stable business identifiers, designing actionable alerts, structuring dashboards along business flows, separating observability responsibilities, halting automation when evidence is unreliable, and using runtime data to continuously correct architectural assumptions.
Palantir's new Object Timeline feature shifts AI agent observability from model-centric traces to business-object lifecycles, surfacing metrics like Agentic Coverage, token usage, runtime, and human wait time to reveal how much work agents actually perform in production versus benchmarks.
This article details how an AI Agent on an orchestration platform automates error code root cause analysis by integrating knowledge bases, code graphs, observability data, and code hosting platforms, achieving 88% consistency with human annotations and zero hard conflicts across 50 test cases through a five-step investigation workflow, dual-path JSON parsing, and regression-tested prompt optimization.
AI AgentAgent OrchestrationError Code Troubleshooting
This article details a practical roadmap for building production-grade fault prediction systems at massive scale, covering target selection, data governance, model evolution, time-window design, and safe action loops—emphasizing that reliable prediction requires engineering rigor beyond just model training.
Alibaba Cloud Monitor MCP exposes nearly 200 tools via Model Context Protocol, letting AI agents query alerts, metrics, traces, and dashboards directly from a remote endpoint — eliminating local CLI setup and cutting a real-world checkout latency investigation from hours to 20 minutes.
This article analyzes three real production incidents caused by broken TraceId propagation, uncontrolled log volume, and sensitive data exposure, then presents a comprehensive microservice logging framework covering log levels, mandatory fields, exception handling, dynamic level governance, cold-hot storage tiering, and a production checklist.
This article details how to build automated fault localization for 10M QPS systems by unifying entity identities, aligning timestamps, integrating change records, and applying a four-layer engine—anomaly normalization, temporal correlation, topological pruning, and causal scoring—to converge millions of anomalies into verifiable hypotheses while avoiding correlation-causation pitfalls through counterfactual evidence and phased rollout.