When CPUs Hide GPU Bottlenecks: How Btune 2.0’s Automated Latency Analysis Breaks the Performance Black Box

1. Hidden Bottleneck: CPU Locks XPU Progress

A real migration of a core inference service from a GPU cluster to a domestically produced XPU cluster resulted in QPS far below theoretical limits and large fluctuations in CPU and XPU utilization. Monitoring revealed abnormal resource usage but could not explain the cause. Cross‑team debugging showed that a Docker deployment performed well, while the same workload run as a K8s pod on the company’s container platform degraded, hinting at a platform‑level issue. By disabling agents one by one, the team isolated a component named halolet. Concurrently, Btune 1.0 hotspot analysis identified the kernel lock _unlocked_loctl as a hotspot. The halolet component repeatedly invoked driver interfaces, holding the kernel lock for extended periods, causing the CPU to wait on the XPU and creating a deadlock‑like state that reduced overall throughput.

2. Btune 2.0: From Resource View to Latency View

Btune 1.0 relied on the USE (Utilization, Saturation, Errors) and TSA (Thread State Analysis) methods, building five bottleneck trees that handled most CPU resource issues. However, AI workloads involve complex CPU‑GPU co‑execution and multi‑process interference, which a pure resource view cannot capture. Btune 2.0 introduces a three‑layer architecture: load profiling + performance diagnosis tree + AI agent . The diagnosis dimensions are expanded to eight areas: CPU, memory, disk, network, GPU/XPU, interconnect, parallelism, and latency, allowing the tool to trace not only where resources are consumed but also where time is spent.

The new deep‑latency analysis module goes beyond application‑level call stacks and inspects kernel execution paths, breaking down:

Scheduler Latency : time a thread spends waiting in the ready queue.

Interrupt/soft‑interrupt preemption: impact of hardware interrupts on normal task execution.

System Call Latency: cost of user‑kernel transitions.

Task Preemption Latency : how high‑priority tasks deprive the current task.

Uninterruptible Wait (D‑state) Latency : precise identification of I/O or lock‑induced process blocking.

This fine‑grained analysis lets developers quickly discern whether a problem is compute‑bound, I/O‑blocked, or caused by synchronization primitives.

The AI agent fuses hardware metrics, a knowledge base, and real‑time profiles, automatically builds multi‑dimensional models, reasons over the diagnosis tree, and invokes appropriate toolchains (e.g., lock analysis, stack collection). It produces two reports: a cost analysis highlighting waste and a performance analysis delivering root‑cause and optimization suggestions, effectively acting as an tireless “chief performance architect.”

3. Automated Practice: Digital‑Human Training Scenario

To validate Btune 2.0 in a complex setting, the tool was applied to a digital‑human model‑training pipeline where developers faced erratic training throughput and overall performance decline. The AI agent performed comprehensive sampling across CPU, GPU/XPU, hardware interconnect, network, disk I/O, and various latency metrics. Initial diagnosis trees ruled out XPU compute and interconnect resource limits, focusing attention on abnormal kernel‑latency data.

Subsequent automated deep‑trace uncovered a pronounced anomaly in the “Uninterruptible Wait (D‑state)” metric. The agent then executed three steps:

Data Collection : continuously scanned the target process within a time window, recording D‑state durations, associated kernel call stacks, and per‑path latency statistics.

Lock Object Identification : automatically flagged lock objects exceeding a threshold, capturing their names and kernel addresses; the case confirmed a lock‑wait abnormality.

Culprit Process Correlation : using the lock’s kernel address, the agent linked all processes contending for the lock, pinpointing the exact “culprit process” that blocked the training job—an insight traditional tools cannot provide.

Compared with conventional single‑process resource views, Btune 2.0 demonstrated powerful cross‑process correlation analysis, automatically generating an explainable performance report without manual call‑chain tracing. Developers applied the report’s recommendations to the culprit process, resulting in a marked improvement in training‑throughput stability.

4. Conclusion

Transitioning from manual debugging to automated diagnosis and from pure resource monitoring to latency‑centric insight, Btune 2.0 reflects a deeper understanding of AI‑infrastructure performance tuning. In an era of soaring compute costs and ever‑larger models, every millisecond saved translates into substantial cost reduction and efficiency gains. Btune 2.0 is positioned not merely as a tool but as a standardized performance‑governance framework, released as open‑source code validated in production, to help developers navigate the complexities of heterogeneous computing.