How HUATUO Builds Kernel Panoramic Observability for Agent Sandboxes

Background and Challenges

In the cloud-native era, container virtualization, shared kernels, resource co-location, and service meshes increase infrastructure complexity, making system failure localization harder. Production incidents often require immediate mitigation — service degradation, container migration, or rebuilds — which destroys the failure scene. Offline reproduction demands massive effort and may still fail to replicate the exact root cause. Modern issues have evolved from periodic, reproducible problems to intermittent, irreproducible ones with extremely short windows and stringent trigger conditions. The industry lacked effective tools for observing kernel-level micro-exceptions.

In the AI era, agent sandbox workloads exhibit high concurrency and bursty resource usage with unpredictable resource profiles. During AI training, hardware becomes a critical resource; any single-point hardware anomaly can slow overall training or cause job failures.

Solution Overview

To achieve system failure analysis, HUATUO targets three goals: system observability, scene traceability, and failure analyzability.

Kernel version compatibility and stability: eBPF makes the kernel programmable and allows safe execution of user-defined code, replacing traditional kernel modules for problem analysis. eBPF combined with kernel dynamic tracing enables zero-intrusion, low-overhead, safe observation probes.

Correlating system and business data: System-layer and application-layer data are naturally isolated, but fault localization requires linking them. By sensing cgroup creation events in the kernel, HUATUO associates container IDs with kernel cgroup addresses; all kernel events carry the cgroup address, which is resolved in user space.

Five-layer system observability: HUATUO combines kprobe, tracepoint, ftrace, perf event, and eBPF dynamic tracing to fetch metrics, events, and call stacks from the host kernel without modifying business code. The capabilities are organized into five layers:

Kernel panoramic metrics: eBPF enables fine-grained kernel metric collection, pinpointing "system slow" to specific subsystems such as scheduler, protocol stack, or BlockIO.

Exception event perception: Hooks on critical system exception paths, slow paths, and normal hooks preserve memory, scheduling, and network exception scenes, preventing evidence loss after recovery.

AutoTracing: Uses sliding windows and threshold triggers to automatically collect flame graphs, kernel stacks, process and file snapshots, balancing low overhead with observation depth.

Continuous performance profiling: Adds a time dimension to traditional flame graphs, enabling historical hotspot retrospection to solve performance jitter.

Heterogeneous hardware awareness: Supports AI compute hardware (ECC, AER, PCIe, GPU/NPU, RoCE) fault perception, improving AI training effective time.

Full-Stack Observability Foundation

Full Network Stack Observation

Implements end-to-end observation from physical link, driver, protocol stack to user space, covering send/receive latency, hardware and software packet loss detection, packet retransmission, and TCP send/receive queue states.

I/O Full-Stack Observation

Centers on inode to trace process-level I/O lifecycle end-to-end across VFS, filesystem, page cache, block queue, driver, and physical device. Rich context correlation locates root causes of slow I/O responses.

CPU Contention Detection

In high-density sandbox nodes, CPU utilization alone cannot reflect contention. HUATUO focuses on internal contention (threads exceed available CPUs within a container), external contention (container competes with other workloads for the runqueue — critical for agent sandboxes where latency may stem from self-concurrency or "noisy neighbors"), and scheduling latency (task runnable but not gaining CPU).

General System Observation

HungTask, Softlockup, IRQ/SoftIRQ latency, and memory reclaim together cover system-level stalls. They detect short-lived anomalies that average utilization misses, such as a CPU holding interrupts disabled for long periods, D-state task pileup, or container direct memory reclaim blocking.

Exception Capture: Capture at Moment, Not Guess Afterwards

Metrics suit trend observation; events suit preserving boundary-defined exception scenes. HUATUO attaches eBPF programs to kernel exception or slow paths. On trigger, kernel-space collection gathers process, call stack, network, or hardware context, passes via Perf Event Buffer to user space for filtering, container correlation, and persistence.

AutoTracing: Automatic Spike Capture

Many performance issues are neither clear kernel exceptions nor reproducible via average metrics (e.g., container CPU spikes for 10 seconds, disk await jitter, anonymous memory rapid growth — all recovered by the time engineers intervene). AutoTracing monitors system anomalies in real time using sliding window, EMA, incremental threshold, or continuous threshold exceedance. Thresholds are not immutable constants; the project provides production-experience defaults but should be calibrated against baselines under different CPU quotas, disk media, and task densities. Flame graphs translate "high CPU" into functions and call paths: left Top Table for quick hotspot symbols, right flame graph for call relationships. For sandbox nodes, targeting can be restricted to VMM PID, thread group, or cgroup to avoid whole-machine aggregation masking a single sandbox's hotspots.

Continuous Performance Profiling: Making Scenes Retrospectable

AutoTracing solves "capture once on anomaly"; continuous profiling solves "how the problem evolves over a period". Traditional ad-hoc sampling aggregates all data into one graph lacking a timeline; continuous profiling retains results per window, turning sporadic slowness into searchable, comparable, retrospectable historical scenes. HUATUO's unified profiler covers On-CPU, Off-CPU, Memory, Block I/O, Lock, etc. Off-CPU profiling is especially suited for "CPU not high but request slow" — it attributes thread off-CPU time to the call path at switch-out, distinguishing blocking wait (I/O, locks, condition variables) from runqueue scheduling wait (CPU contention). Native memory profiling separates three often-conflated concepts: virtual_alloc (virtual address space requested), physical_alloc (physical pages allocated during window), physical_usage (physical pages resident at collection time). An image shows physical memory residency aggregated by process and call path, drilling from "node memory high" to specific allocation paths.

For single-machine scenes, a CLI is available. On the platform side (huatuo-apiserver), the Profiling API supports capability query, task creation, status/result viewing, raw data retrieval, stop, and delete — evolving ad-hoc commands into a servicized, task-based continuous profiling platform.

From RAS to GPU/NPU

In AI infrastructure, hardware issues don't always appear as device offline. Rising ECC correctable errors, PCIe link width degradation, AER errors, GPU throttling, or RoCE retries may first manifest as throughput drop or tail latency. HUATUO captures structured hardware events via Linux RAS: MCE, EDAC, ACPI GHES, PCIe AER, feeding alerting and audit pipelines. This serves four targets: early warning (rising ECC CE frequency on same DIMM triggers proactive replacement), fault isolation (GPU, HCA, PCIe, NIC link anomalies enable node isolation and task migration), performance degradation localization (correlating NVMe/HBA AER with I/O latency to distinguish software, device, and link issues), and audit retrospection (retaining timestamp, device BDF, severity, raw fields for traceable evidence). The key is correlating hardware events with scheduling, network, I/O, and profiling data in the same time window so the platform can judge whether an agent stall originates from a software call path or underlying device degradation.

Agent Sandbox Hierarchical Correlation

After probes collect kernel stacks, the question remains "whose scene is this?" Container environments use cgroup CSS, network namespaces, and container runtime metadata for correlation; microVM sandboxes additionally require Sandbox-to-VMM mapping.

Community Progress

HUATUO was open-sourced by DiDi in 2025 and donated to CCF. As of August 2026, v2.3.0 has ~1.1k GitHub Stars, 51 contributors, 1,310 commits, image pulls exceeding 15k, and deployments in 20+ enterprises. MooXi and Huawei Ascend communities contribute GPU/NPU capabilities. Recognitions include CCF-Guanghua Youth Open Source Special Fund Seed Key Project, entry into CNCF Landscape, and listing as an emerging project by the eBPF Foundation. Future directions: MCP fault analysis (standard tool interface for Copilot/Agent), distributed trace analysis (extending single-node kernel scenes to cross-node network and task chains), and AI scenario performance analysis (deepening GPU/NPU training/inference software-hardware co-analysis).