Why Is GPU Utilization Low? Try This Zero‑Intrusion AI Profiling Tool

SysOM AI Profiling is a component of the Alibaba Cloud operating system console that provides zero‑intrusive, full‑lifecycle performance observation and diagnosis for AI jobs, covering training and inference, single‑card to thousand‑card clusters, and Python to GPU kernel layers.

Existing tools can capture fine‑grained CUDA streams, kernels, or framework‑level metrics, but they are often highly invasive, suited only for short‑term offline analysis, tightly coupled to specific runtimes or framework versions, and usually cover a single dimension, making cross‑vendor compatibility and comprehensive diagnosis difficult.

To overcome these limitations, the next‑generation AI Profiling tool must be zero‑intrusive, cover the entire lifecycle, and be cross‑vendor compatible.

Core Features

Key Profiling (minutes‑level results): Input an instance ID and GPU process; the tool automatically collects, uploads, and analyzes data without code changes, container restarts, or manual trace handling. Results are displayed directly in the front‑end timeline view.

Multi‑dimensional Deep Observation: Simultaneously captures Python call stacks, CPU usage, GPU operators, Torch calls, memory usage, FLOPS, RDMA communication, TCP traffic, etc., allowing both macro‑level time distribution and micro‑level GPU kernel analysis.

Two View Modes:

Multi‑process aggregation view – shows process list, GPU IDs, memory usage, and entry points for per‑process reports.

Single‑process view – displays device and memory overview, GPU utilization, stage time ratios, GPU operator breakdown, and detailed kernel statistics.

Memory Snapshot Analysis: Shows active memory allocation/release timelines for each memory block, enabling pinpointing of abnormal memory growth and drilling down to the Python call stack.

Iteration‑anchored Differential Analysis: For large‑scale AI training/inference, the tool marks iteration boundaries, aggregates metrics (loss, compute time, storage, communication) per iteration, and visualizes outlier iterations (e.g., gradient explosion or high communication latency).

Zero Intrusion: No container modifications or restarts required.

Cross‑vendor Compatibility: Supports Nvidia, PPU, AMD devices.

Minimal Overhead: After profiling ends, performance quickly returns to pre‑profiling levels.

Rich Data Collection: On‑demand collection of Python stacks, CPU info, GPU operators, Torch, memory, FLOPS, RDMA, TCP, and optional GPU/CPU monitoring.

Full‑process Automation: Minutes‑level end‑to‑end automation from instance selection to report generation, with built‑in timeline view (no external Perfetto/Chrome tracing needed).

Multi‑process Aggregation: Supports concurrent process collection and merged GPU kernel analysis.

Multi‑dimensional Capture: Supports time‑based and iteration‑based collection, custom iteration entry points, and skipping initial iterations.

Industry‑level Advantages: In addition to GPU kernel and Torch data, it captures RDMA, system calls, DCGM metrics, NVTX tags, and supports both Python and C++ processes, providing unified monitoring for non‑GPU processes as well.

Case Study 1 – vLLM Inference Memory Leak

Problem: A vLLM service experienced OOM despite pre‑allocated memory, with observable memory growth during runtime.

Solution: Using AI Profiling, the timeline revealed that memory spikes coincided with cudaMalloc calls. Drilling into the Python call stack identified the framework’s context‑management logic that dynamically reserved extra memory during request handling, causing the leak.

Optimization Recommendations:

Avoid memory fragmentation by setting CUDA_PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb to a lower value.

Use torch.cuda.empty_cache() or adjust model/data loading strategies to reduce cached blocks.

Case Study 2 – Performance Comparison Between SGLang and vLLM

Using the same profiling workflow, the tool compared SGLang and vLLM pipelines.

Image encoding in SGLang lacks batching, leading to higher per‑request latency.

Text decoding shows similar latency across both, with batch size having little impact.

GPU idle periods stem from insufficient kernel dispatch at stage starts and occasional millisecond‑level gaps likely caused by OS activity.

GPU gaps during text decode are mainly due to CUDA Graph overhead.

These observations helped identify optimization targets for both frameworks.

Profiling Agent – From Observation to Automated Diagnosis

Building on AI Profiling’s data collection, the Profiling Agent adds source‑level analysis, automatically interpreting metrics, locating root causes, and generating remediation suggestions, forming a closed‑loop "sense → diagnose → analyze → fix" workflow. The agent is already released and will be detailed in future posts.

Overall, AI Profiling delivers zero‑intrusive, low‑overhead, cross‑vendor performance insight across the entire AI workload lifecycle, turning previously opaque GPU utilization, memory anomalies, and communication tail issues into traceable, actionable information, validated by real‑world vLLM memory‑leak diagnosis and SGLang vs. vLLM performance studies.