Tagged articles
GPU utilization
31 articles · Page 1 of 1
This article explains why GPUs underutilize during LLM inference and compares static, dynamic, and continuous batching strategies, detailing their trade-offs, the role of KV cache, iteration-level scheduling, and chunked prefill to optimize throughput for variable-length outputs.
GPU utilizationKV CacheLLM inference
0 likes · 13 min read

The article introduces SysOM AI Profiling, a zero‑intrusive, cloud‑native performance observation and diagnosis solution for AI workloads that spans training to inference, single‑GPU to multi‑GPU clusters, and Python to GPU kernel layers, helping users pinpoint low GPU utilization, memory leaks, and communication bottlenecks.
AI profilingGPU utilizationPerformance Analysis
0 likes · 14 min read

In large‑model training, a smooth loss decline is only a necessary condition; the article explains which additional logs—learning‑rate, gradient norm, parameter updates, validation metrics, throughput, GPU usage, and checkpoint status—must be monitored and how to diagnose common issues with concrete code examples.
GPU utilizationPyTorchcheckpointing
0 likes · 22 min read

Low GPU utilization and mysterious memory growth often stall AI workloads, but existing tools are intrusive and limited; SysOM AI Profiling offers zero‑intrusion, one‑click, multi‑dimensional analysis across training and inference, with timeline visualisation, memory‑snapshot, and iteration‑anchored diff, demonstrated by vLLM memory‑leak detection and SGLang vs vLLM performance comparison.
AI profilingGPU utilizationMemory Leak
0 likes · 11 min read

The article explains continuous batching, a scheduling technique that re‑evaluates the batch at every inference iteration, eliminating the straggler problem of static batching, and shows how it, together with PagedAttention, can raise LLM service throughput dramatically while balancing latency and memory usage.
GPU utilizationLLM inferencecontinuous batching
0 likes · 9 min read

The article presents a comprehensive AI FinOps framework that attributes every AI expense to specific apps, users, and tasks, normalizes diverse cost units, and applies token economics, smart routing, semantic caching, and budget controls to ensure sustainable AI operations and measurable ROI.
AI FinOpsCost attributionGPU utilization
0 likes · 33 min read

The article explains how Tencent CodeBuddy’s AI DLC platform unifies Spark batch processing and Ray training to eliminate data movement, turning agent trajectories into reusable training fuel, which reduces monthly‑level training cycles to weekly, enables in‑place computation on billions of features, and cuts operational costs by 60%.
AI DLCGPU utilizationRay
0 likes · 2 min read

This article examines why traditional software testing fails for large language models, outlines common misconceptions, introduces a four‑dimensional LATC metric framework, and provides a detailed, step‑by‑step case study and engineering pipeline for reliably measuring and improving LLM latency, throughput, availability, and cost.
GPU utilizationLLMlatency
0 likes · 8 min read

xAI’s massive fleet of roughly 550,000 Nvidia H100 and H200 GPUs in its Memphis and Colossus data centers is operating at a mere 11% model FLOPs utilization, highlighting how scaling to hundreds of thousands of GPUs creates coordination, network, and scheduling bottlenecks that waste most of the hardware’s compute power.
AI infrastructureGPU utilizationNVIDIA H100
0 likes · 5 min read

DeepSeek V4 achieves ultra‑long context, complex training pipelines, and custom high‑performance kernels by enforcing batch invariance, a design that guarantees bit‑wise identical outputs across varying batch shapes but incurs lower GPU utilization, reduced small‑batch speed, and added engineering complexity.
Batch InvarianceGPU utilizationLLM Engineering
0 likes · 8 min read

The article reveals that the perceived AI‑GPU shortage stems from misleading utilization metrics and wasted capacity, not actual supply constraints, and argues that better measurement and orchestration—not buying more hardware—will determine competitive advantage in the emerging AI infrastructure market.
GPU utilizationindustry analysisorchestration
0 likes · 9 min read

The article details how Huolala’s Dolphin platform engineers large‑model inference for high‑concurrency, long‑context, low‑latency production workloads, achieving 50‑60% GPU cost reduction through systematic resource allocation, model quantization, PD‑separation, speculative sampling, and kernel‑level optimizations while maintaining service stability.
GPU utilizationLarge Model InferenceResource Optimization
0 likes · 18 min read

This article explains how Alluxio’s distributed caching architecture tackles the massive, multimodal data challenges of embodied AI, delivers near‑zero‑millisecond access, achieves 99.57% GPU utilization in MLPerf Storage v2.0, and validates its value through real‑world enterprise deployments.
AI data platformAlluxioData Infrastructure
0 likes · 21 min read

The article details the Attention‑FFN Disaggregation (AFD) technique used by Baidu Baige to separate self‑attention and feed‑forward network stages in DeepSeek‑V3 models, describing multi‑stage scheduling, three‑batch overlap, communication optimizations, and performance results that achieve up to 19% throughput improvement under a 100 ms SLO.
3BOAFDAttention-FFN Disaggregation
0 likes · 17 min read

Token‑level Two‑Chunk Overlap replaces traditional batch‑level Two‑Batch Overlap, dynamically splitting sequences into balanced token chunks, enabling near‑equal compute and communication times, improving GPU utilization and achieving up to 30% throughput gains in heterogeneous request workloads, with zero accuracy loss.
GPU utilizationLLM inferenceSGLang
0 likes · 9 min read

Alluxio’s distributed caching dramatically accelerates AI training and checkpointing workloads, achieving up to 99.57% GPU utilization and linear scaling across clusters in the MLPerf Storage v2.0 benchmark, while using cost‑effective commodity hardware to eliminate I/O bottlenecks.
AI TrainingAlluxioDistributed Caching
0 likes · 11 min read

This article details the Angel‑HCF team's end‑to‑end DeepSeek inference optimizations—including PD separation, multi‑layer MTP, EP and DP parallelism, hardware‑aware kernels, and load‑balancing strategies—that boost throughput to over 15,800 tokens per second while keeping per‑token latency under 50 ms.
AI performanceDeepSeekGPU utilization
0 likes · 13 min read

This article analyzes common GPU efficiency problems in enterprise AI compute platforms—such as low utilization, long fault‑resolution times, and limited performance gains—and presents three practical solutions: dynamic resource allocation, systematic fault‑tolerance, and system‑level tuning, illustrated with real‑world case studies.
AI PlatformGPU utilizationlarge model training
0 likes · 11 min read

Large‑model inference engines convert prompts into responses via a Prefill stage and an autoregressive Decoder, measured by TTFT and TPOT, and Baidu’s AIAK suite improves TPOT by separating tokenization, using static slot scheduling, and asynchronous execution, cutting token‑interval latency from ~35 ms to ~14 ms and boosting GPU utilization to about 75 % while also leveraging quantization and speculative execution for higher throughput.
AI accelerationGPU utilizationLarge Model Inference
0 likes · 10 min read

This article explains the architecture of large‑model inference engines, key performance metrics like TTFT and TPOT, the limitations of popular engines such as vLLM, and Baidu Baige's AIAK solutions—including multi‑process, static slot, and asynchronous execution—that dramatically reduce token‑interval latency and increase GPU utilization.
AIAKGPU utilizationLLM performance
0 likes · 10 min read

eBay upgraded its AI infrastructure by adopting Ray, cutting model development and deployment time by roughly 50% and boosting GPU utilization from about 10% to over 75% through automated cluster scaling and high‑throughput batch inference.
AI infrastructureData+AIGPU utilization
0 likes · 5 min read

With the growing complexity of LLM architectures like GQA, MLA, and MoE, runtime overhead has become a bottleneck; this article analyzes Python performance, communication costs, and synchronous execution in current inference frameworks, introduces the fully asynchronous TAG architecture, and demonstrates its superior throughput and latency through benchmarks.
GPU utilizationLLM inferenceRuntime Optimization
0 likes · 12 min read

The article outlines the challenges of massive AI training data, defines storage performance requirements, and presents Polefs—a cloud‑native distributed cache file system with unified storage, metadata acceleration, and read/write caching designed to improve GPU utilization and reduce data redundancy.
AIGPU utilizationPoleFS
0 likes · 14 min read

GoldMiner, a new system from Alibaba Cloud’s PAI platform, elastically scales deep learning data pre‑processing pipelines, dramatically improving training performance up to 12.1× and GPU cluster utilization by 2.5×, and its underlying research was accepted at SIGMOD 2023.
GPU utilizationSIGMODdata preprocessing
0 likes · 5 min read

Large-model offline (batch) inference, which processes massive data on billion-parameter models, faces GPU memory and distributed scheduling challenges; this article explains how Ray's cloud-native framework, model parallelism, and Ray Datasets pipelines address these issues, improve throughput, and enable elastic, efficient GPU utilization.
GPU utilizationLarge Model InferenceRay
0 likes · 16 min read

Bilibili’s self‑developed InferX framework, combined with NVIDIA Triton Inference Server, streamlines AI model serving by adding quantization, structured sparsity, and custom kernels, delivering up to eight‑fold throughput gains, cutting GPU usage by half, and enabling faster, cost‑effective OCR and large‑model deployments.
AI inferenceGPU utilizationInferX
0 likes · 10 min read

The article explains how AIGC models demand massive GPU compute, why network bandwidth and latency become the critical limiting factors, and how the Distributed Disaggregated Chassis (DDC) architecture addresses these challenges with scalable, high‑throughput networking solutions.
AI infrastructureAIGCDDC
0 likes · 13 min read

The article presents high_service, a high‑performance inference framework that boosts GPU utilization in multimedia AI services by separating CPU‑heavy preprocessing from GPU inference, employing priority‑based auto‑scaling, multi‑tenant sharing, and TensorRT‑accelerated models to eliminate GIL bottlenecks, reduce waste, and adapt to fluctuating traffic, with future work targeting automated bottleneck detection and further CPU‑GPU offloading.
Auto ScalingGPU utilizationHigh-Performance Computing
0 likes · 19 min read


ITPUBApr 27, 2022 · Artificial Intelligence
This article details the design and optimization of 58.com’s WPAI machine learning platform, covering background, training‑task scheduling, elastic inference scaling, offline‑online resource mixing, and model‑inference acceleration, and shows how these techniques collectively raised GPU usage by 51% and CPU usage by 38% while cutting costs.
AI PlatformElastic ScalingGPU utilization
0 likes · 26 min read

The article outlines a technical exchange hosted by 58.com AI Lab and Tianjin University that discusses high‑efficiency AI computing, resource‑aware scheduling for both online inference and offline training, and methods to mitigate GPU under‑utilization and gray‑interference in distributed deep‑learning platforms.
AIGPU utilizationinference
0 likes · 4 min read

A detailed case study shows how a mobile game's sudden frame‑rate drop on a Galaxy S9+ was traced to thermal throttling, highlighting the importance of correlating CPU, GPU, memory, and temperature data during performance testing.
CPU frequencyGPU utilizationframe rate
0 likes · 5 min read
