Tagged articles
GPU monitoring
11 articles · Page 1 of 1
Comprehensive guide covering PromQL fundamentals, node/container/GPU/business metrics, alerting anti-patterns, and production-grade rule libraries with real troubleshooting cases.
GPU monitoringKubernetesObservability
0 likes · 64 min read

This article details building production-grade GPU cluster observability using DCGM Exporter and Prometheus, covering hardware counter metrics (SM_ACTIVE, DRAM_ACTIVE), Kubernetes pod mapping, MIG monitoring, alerting rules, and real-world troubleshooting cases like NVLink degradation and thermal throttling.
AI infrastructureDCGM ExporterGPU monitoring
0 likes · 64 min read

A comprehensive guide to building a three-layer observability stack for vLLM inference services, covering GPU metrics (DCGM), engine-level queue and KV cache metrics, request-level latency (TTFT/TPOT), Prometheus alerting rules, Grafana dashboard design, and troubleshooting runbooks with real-world case studies.
DCGMGPU monitoringGrafana
0 likes · 71 min read

This article provides a step‑by‑step, Linux‑focused workflow for recording driver and GPU versions, interpreting utilization versus memory metrics, locating memory‑consuming processes, handling container and Kubernetes mappings, checking temperature, power, ECC, MIG, driver health, OOM conditions, and setting up reliable monitoring and alert thresholds for data‑center GPUs.
CUDAGPU monitoringKubernetes
0 likes · 28 min read

This guide explains how to continuously monitor NVIDIA GPU utilization, memory, temperature, power and error metrics using DCGM Exporter, covering driver verification, Docker and Compose deployment, Prometheus scraping, Kubernetes DaemonSet setup, custom collectors, PromQL queries, alert rules and troubleshooting procedures.
DCGM ExporterDockerGPU monitoring
0 likes · 28 min read

Huatuo, the open‑source deep‑observability platform backed by Didi, now supports real‑time monitoring of MetaX GPUs, offering detailed hardware metrics via Docker or Kubernetes deployments and exposing them through a /metrics endpoint for cloud‑native AI and operations use cases.
AI infrastructureGPU monitoringHuatuo
0 likes · 4 min read

This guide introduces nvitop, an interactive NVIDIA‑GPU process viewer and resource manager, explains its key features, shows how to install it via uvx/pipx, demonstrates basic device and process commands as well as the real‑time monitoring mode, and provides troubleshooting tips for common issues.
GPU monitoringLinuxNVIDIA
0 likes · 5 min read

The article introduces GPUprobe, an eBPF‑based tool that provides lightweight, continuous, application‑level monitoring of CUDA memory allocation, leaks, and kernel launches, compares it with NSight Systems and DCGM, and demonstrates near‑zero overhead integration with Prometheus and Grafana through detailed code examples and real‑world output analysis.
GPU monitoringGrafanaObservability
0 likes · 13 min read

This article systematically reviews GPU monitoring for large‑scale AI training, covering MFU/HFU definitions, key DCGM metrics, NVLink bandwidth, common failure codes such as Xid and SXid, experimental insights on T4 and H100 GPUs, and practical case studies for diagnosing and mitigating performance drops.
DCGMGPU failuresGPU monitoring
0 likes · 26 min read

This guide explains how to install and use three command‑line tools—lm‑sensors, Glances, and i7z—on Ubuntu to monitor CPU and GPU temperatures, fan speeds, and other hardware metrics, providing step‑by‑step commands and example outputs for effective laptop cooling diagnostics.
CPU temperatureGPU monitoringGlances
0 likes · 6 min read

This article explains how to use TensorBoard and the Timeline tool to monitor TensorFlow GPU utilization, identify operation bottlenecks, and visualize execution times, including code examples and steps for exporting and merging profiling data for repeated runs.
GPU monitoringTensorBoardTensorFlow
0 likes · 7 min read
