Why Is GPU Utilization Low and Hard to Diagnose? Try This Zero‑Intrusion AI Profiling Tool

Problem Statement

Many AI training or inference jobs finish with GPU utilization below 30% or experience unexplained memory growth, making root‑cause diagnosis difficult.

Limitations of Existing Tools

Current tools either capture fine‑grained CUDA streams, monitor at the pod level, or use eBPF, but they are intrusive, suited for short‑term offline analysis, tightly coupled to specific frameworks or hardware, and often provide only a single dimension of data.

Design Goals of SysOM AI Profiling

Zero‑intrusion, full‑lifecycle coverage, and cross‑vendor compatibility.

Core Capabilities

One‑click profiling: provide instance ID and process, the system automatically collects, uploads, and analyses data without code changes or restarts.

Multi‑dimensional observation: collects Python call stacks, CPU metrics, GPU kernels, Torch layer calls, memory usage, FLOPS, RDMA, TCP, etc.

Aggregated and per‑process views with timeline visualisation.

Memory‑snapshot analysis and active‑memory timeline.

Iteration‑anchored differential analysis to spot outlier iterations.

Case Study 1 – vLLM Inference Memory Leak

Background: a vLLM service ran out of memory despite pre‑allocated buffers.

Navigate to the AI Profiling page, enter the instance ID and start profiling.

The system collected data and generated a report.

GPU Kernel timeline showed a peak at a cudaMalloc call; drilling into the Python stack identified dynamic memory reservation in the request‑handling logic.

Root cause: the framework’s context manager allocated extra memory during inference, causing OOM.

Recommendations: reduce

CUDA_PYTORCH_CUDA_ALLOC_CONF

max_split_size_mb

to avoid fragmentation, and consider torch.cuda.empty_cache() or adjust model/data loading.

Case Study 2 – SGLang vs vLLM Performance Comparison

Using the same profiling workflow, the tool compared two inference engines.

SGLang: image encoding not batched, higher per‑request latency; text decoding batch size has little impact; GPU idle time caused by insufficient kernel launch at stage start and occasional OS‑related gaps; decode stage GPU gaps stem from CUDA Graph overhead.

vLLM pipeline: slightly higher image encoding latency, comparable decode latency.

Future Direction – Profiling Agent

Building on the data collection layer, the upcoming Profiling Agent will automatically interpret the observations, locate root causes, and suggest fixes, forming a closed “sense‑diagnose‑analyze‑remediate” loop.