LLM Inference Monitoring: GPU Utilization, Throughput & Latency Tracking for vLLM

Problem Background: Why nvidia-smi Shows 98% but Service Is Slow

The article opens with a real scenario: an 8×H100 node running vLLM serving Qwen models sees P99 TTFT jump from 800 ms to 6 s. The on-call engineer runs nvidia-smi and sees GPU-Util 100%, memory 78/80 GB, concluding "GPU saturated, add more cards." That conclusion is usually wrong. GPU-Util in nvidia-smi measures the fraction of time any kernel runs on the GPU, not how much useful compute occurs. A tight loop launching a tiny memcpy every 10 µs also drives GPU-Util to 100%. Three common high-utilization/low-efficiency states in inference:

KV-cache saturation causing scheduler spin : vLLM's block manager cannot allocate new blocks; the scheduler preempts requests, releases their KV, and re-queues them. GPU stays busy recomputing prefill instead of decoding. Symptoms: GPU-Util 100%, vllm:num_requests_waiting rising, e2e_request_latency_seconds worsening, but generation_tokens_total growth rate drops.

Memory-bandwidth bottleneck on small batches : Decode reads all weights from HBM per token; small batch ⇒ low arithmetic intensity. DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (tensor-core active cycles) may be 5–15% while DCGM_FI_DEV_GPU_UTIL is 90%+. The gap is the "memory-bound" fingerprint.

GPU-CPU synchronization hotspots : Sampling, logits post-processing, or streaming callbacks contain synchronous ops; GPU waits on CPU each step. Manifests as sawtooth GPU-Util.

Therefore "GPU utilization" must be split into at least three metrics: time-based occupancy ( GPU_UTIL), compute-unit activity ( PIPE_TENSOR_ACTIVE / SM_ACTIVE), and memory-bandwidth saturation ( DRAM_ACTIVE). Watching only the first is like judging fuel economy by RPM alone.

Three-Layer Metric Architecture

Observability is organized in three layers, each answering a different question:

┌──────────────────────────────────────────────────────────────┐
│  Request Layer (user view)                                   │
│  TTFT / TPOT / E2E latency / success rate / token dist.      │
│  Answers: What does the user feel?                           │
└──────────────────────────┬───────────────────────────────────┘
                           │ Request-layer degradation explained by lower layers
┌──────────────────────────▼───────────────────────────────────┐
│  Engine Layer (scheduler view)                               │
│  running/waiting queues / KV cache usage / preemption        │
│  prefix-cache hits / token throughput (prompt, generation)   │
│  Answers: What is the engine doing? Why queuing?             │
└──────────────────────────┬───────────────────────────────────┘
                           │ Engine bottlenecks ultimately land on hardware
┌──────────────────────────▼───────────────────────────────────┐
│  GPU Layer (hardware view)                                   │
│  GPU_UTIL / TENSOR_ACTIVE / SM_ACTIVE / DRAM_ACTIVE / VRAM   │
│  temp / power / freq / ECC / XID                             │
│  Answers: Is hardware busy or idle? Busy doing what?         │
└──────────────────────────────────────────────────────────────┘

Troubleshooting order must be top-down : confirm slowness at request layer, determine queue vs. execution at engine layer, then identify compute vs. bandwidth at GPU layer. Starting with GPU-Util leads to false-busy distractions.

Precise Latency Definitions

TTFT ( vllm:time_to_first_token_seconds): request arrival → first output token. Includes queue time + prefill + first sampling. Histogram from scheduler entry; gateway queuing not included.

TPOT ( vllm:request_time_per_output_token_seconds): (e2e_latency - ttft) / (output_tokens - 1). Denominator subtracts 1 because first token is in TTFT. Single-token requests record 0. User-perceived typing speed = 1/TPOT.

ITL ( vllm:inter_token_latency_seconds): interval between consecutive streamed tokens; one sample per token. TPOT is average; ITL is per-step. User complaints of "stutter" map to ITL P99 spikes while TPOT mean looks normal. Both needed.

E2E ( vllm:e2e_request_latency_seconds): scheduler entry → last token done.

Relationship: E2E ≈ Queue + Prefill + (OutputTokens - 1) × TPOT, where Queue + Prefill ≈ TTFT. If E2E degradation cannot be explained by this equation, check http_request_duration_seconds for gateway/client-side delays.

Two Kinds of Throughput

Request throughput (QPS) : rate(vllm:request_success_total[1m]) split by finished_reason (stop, length, abort). Only stop+length represent completed work.

Token throughput : input rate(vllm:prompt_tokens_total[1m]), output rate(vllm:generation_tokens_total[1m]). Must be separate because prefill is compute-bound, decode is memory-bound. Summing them masks workload differences: long-prompt/short-answer (RAG) vs. short-prompt/long-answer (chat) can have identical total token throughput but vastly different GPU pressure.

Effective vs. nominal throughput : effective = nominal × success rate, accounting for aborted requests wasting compute.

GPU Metric Meanings & Normal Ranges

DCGM_FI_DEV_GPU_UTIL (DCGM/NVML): Fraction of sampling period with any kernel running. Normal: Decode often 80–100%, but diagnostically useless alone.

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (DCGM profiling): Tensor-core pipe active cycles. Normal: Prefill 30–60%, decode often <20%.

DCGM_FI_PROF_SM_ACTIVE (DCGM): Cycles with ≥1 warp resident on SM. Normal: Usually near GPU_UTIL; large gap = scheduling overhead.

DCGM_FI_PROF_DRAM_ACTIVE (DCGM): Memory-controller active cycles. Normal: Decode >50% ⇒ memory-bound.

DCGM_FI_DEV_FB_USED (DCGM/NVML): Used frame-buffer (MiB). Normal: Stable near gpu_memory_utilization × total VRAM.

DCGM_FI_DEV_POWER_USAGE (DCGM): Power draw (W). Normal: H100 PCIe 350 W / SXM 700 W ceiling.

DCGM_FI_DEV_SM_CLOCK (DCGM): SM clock (MHz). Normal: Varies with load and power limit.

DCGM_FI_DEV_XID_ERRORS (DCGM): Latest XID error code. Normal: Should be 0. DCGM_FI_PROF_* metrics have higher collection overhead and require profiling permissions. GPU Operator defaults vary; verify with --collectors.

Rule of thumb: judge health by generation_tokens_total growth rate per GPU (tokens/sec/GPU), then TTFT/TPOT percentiles, then GPU utilization. High util + low per-GPU tokens = waste.

Five Easily Confused "Utilizations" & Decision Matrix

Device Util ( DCGM_FI_DEV_GPU_UTIL): Numerator = Time with kernel running. High means: Card not idle, that's all.

SM Active ( DCGM_FI_PROF_SM_ACTIVE): Numerator = Cycles with ≥1 warp resident. High means: Warp-scheduler saturated.

Tensor Core Active ( DCGM_FI_PROF_PIPE_TENSOR_ACTIVE): Numerator = Cycles executing tensor instructions. High means: Actually doing matmuls.

DRAM Active ( DCGM_FI_PROF_DRAM_ACTIVE): Numerator = Memory-controller busy cycles. High means: Stalled on data movement.

KV Pool Usage ( vllm:kv_cache_usage_perc): Numerator = Allocated blocks. High means: Scheduler context capacity exhausted.

Combined decision matrix (GPU_UTIL, TENSOR_ACTIVE, DRAM_ACTIVE, KV_Usage):

High GPU_UTIL, High TENSOR_ACTIVE, Med DRAM_ACTIVE, Med KV Usage → Healthy prefill-heavy load

High GPU_UTIL, Low TENSOR_ACTIVE, High DRAM_ACTIVE, Low-Med KV Usage → Decode memory-bound, small batch — normal but optimizable

High GPU_UTIL, Low TENSOR_ACTIVE, Med DRAM_ACTIVE, ≈1.0 KV Usage → KV full → preemption/recompute — unhealthy

Low GPU_UTIL, Low TENSOR_ACTIVE, Low DRAM_ACTIVE, Low KV Usage → No traffic, or scheduler idle / CPU stuck

Low GPU_UTIL, High TENSOR_ACTIVE, High DRAM_ACTIVE, Med KV Usage → Sampling or GPU-CPU sync wait; brief bursts normal

Collection Pipeline

┌─────────────┐   /metrics (9400)   ┌────────────┐
│ dcgm-exporter│────────────────────▶│            │
└─────────────┘                     │ Prometheus │──▶ Grafana
┌─────────────┐   /metrics (8000)   │  (scrape)  │
│ vllm server │────────────────────▶│            │──▶ Alertmanager
└─────────────┘                     │            │
┌─────────────┐   /metrics (9100)   │            │
│node-exporter│────────────────────▶│            │
└─────────────┘                     └────────────┘
┌─────────────┐   /metrics (8080)
│ vllm-router │───────────────────────────────────┘
└─────────────┘

Four scrape targets: dcgm-exporter (hardware), vllm server (engine), node-exporter (host NIC, CPU, filesystem — especially NFS/object-store mount latency), vllm-router (routing queue if present).

First Step: Dump Real Metrics

Never write PromQL from memory. Before building dashboards, pull live metric catalogs:

# vLLM
curl -s http://10.0.10.21:8000/metrics | grep '^# HELP' | awk '{print $3}' | sort
# DCGM
curl -s http://10.0.10.21:9400/metrics | grep -E '^DCGM_FI_(DEV_GPU_UTIL|DEV_FB_USED|PROF_PIPE_TENSOR_ACTIVE|PROF_DRAM_ACTIVE)' | head -20

Verify labels ( modelName, UUID, Hostname, gpu, device) for alignment. Cross-check nvidia-smi vs. DCGM_FI_DEV_FB_USED; mismatch suggests MIG instance confusion or stale cache.

Scrape Intervals & Overhead

vLLM /metrics cheap (in-memory counters); 15 s interval fine. Watch scrape_samples_post_metric_relabeling as series count grows with GPU/model count.

DCGM PROF metrics controlled by --collect-interval (default 30 s). Don't confuse with Prometheus scrape_interval; scraping faster than exporter updates just duplicates samples.

Each /metrics request serializes all series; large max_num_seqs + many histogram buckets ⇒ hundreds of KB per scrape. If exporter co-locates with inference, CPU contention can jitter prefill.

DCGM Exporter Deployment

Bare-metal systemd

Install DCGM: apt-get install -y datacenter-gpu-manager-4-cuda12; enable nvidia-dcgm.

Download exporter (pin version, e.g., 4.4.2-4.6.0).

Create /etc/dcgm-exporter/metrics.csv whitelist (field, type, description — three columns). Example includes DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_SM_CLOCK, DCGM_FI_DEV_XID_ERRORS, DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE. Whitelist cuts 30–50% serialization overhead.

Systemd unit with -c 30000 (30 s collect), -d 1000 (1 s stats window), --web.listen-address=:9400.

Validate: curl -s localhost:9400/metrics | grep -c '^DCGM_FI_' should match enabled fields × GPU count.

Kubernetes (GPU Operator)

Operator ≥23.x deploys nvidia-dcgm-exporter DaemonSet on port 9400. Don't double-deploy. For custom whitelist, use ConfigMap + ClusterPolicy.dcgmExporter.config.name (not direct DaemonSet patch, which Operator upgrades overwrite).

Prometheus Scrape Config

Bare-metal: static_configs with relabel to extract instance_ip and instance. K8s: ServiceMonitor selecting app.kubernetes.io/name=nvidia-dcgm-exporter; verify Status → Targets shows endpoint count = GPU nodes.

vLLM Metrics Exposure & Labeling

Startup Flags

vllm serve /models/Qwen3-32B \
  --served-model-name qwen3-32b \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --port 8000

Don't confuse --port / --host. --host 0.0.0.0 exposes metrics + API on LAN; restrict port 8000 via firewall/nftables to Prometheus/gateway CIDRs only.

Multi-replica Label Design

Three replicas share model_name label; differentiate via Prometheus target labels ( instance, pod). Deployment adds model: qwen3-32b, team: ai-platform pod labels; PodMonitor with jobLabel: model turns pod label into job="qwen3-32b" for easy Grafana selection. Set CPU/memory limits (e.g., 32 GiB, 16 CPU for 32B TP=4) to avoid OOM killing neighbors.

Custom Histogram Buckets

Default buckets too coarse for SLO-critical regions. vLLM 0.32+ supports --custom-histogram-buckets:

--custom-histogram-buckets '{"vllm:time_to_first_token_seconds": [0.05,0.1,0.2,0.3,0.5,0.75,1.0,2.0,5.0,10.0], "vllm:request_time_per_output_token_seconds": [0.005,0.01,0.02,0.03,0.05,0.08,0.12,0.2,0.5]}'

Verify flag format per version ( vllm serve --help | grep -A3 custom-histogram). If unsupported, fall back to default buckets and note quantile error margins.

Multi-process Pitfall

--api-server-count >1

hides process_resident_memory_bytes, process_cpu_seconds_total (default Python registry). Alerts on vLLM process memory silently stop firing. Fix: use cAdvisor container_memory_working_set_bytes or sidecar scraper; add to alert self-check checklist.

Prometheus Alerting Rules

Request Layer: SLO-based

Recording rules for vllm:ttft_p95:5m, vllm:ttft_p99:5m, vllm:tpot_p95:5m, vllm:request_success:rate5m (split by finished_reason). Alerts: VllmHighTTFT (P95 > 1.5 s for 10 m), VllmHighTPOT (P95 > 0.15 s for 10 m). Thresholds derived from load-test baselines, not copied.

Engine Layer: Queue & KV Cache

VllmQueueBacklog

( max(vllm:num_requests_waiting) > 20 for 5 m), VllmKVCacheNearlyFull ( max(vllm:kv_cache_usage_perc) > 0.95 for 10 m), VllmPreemptionSpike ( rate(vllm:num_preemptions_total[5m]) > 0.2 for 10 m — verify metric exists per version), VllmAbortRatioHigh (abort rate > 15% for 15 m).

GPU Layer: Distinguish "Busy" from "Effectively Busy"

Recording rule gpu:tensor_active:p50. Alert GpuUtilHighButTensorLow: avg(GPU_UTIL) > 85 and avg(TENSOR_ACTIVE) < 0.20 for 15 m. Critical hardware alerts: GpuXidError ( XID_ERRORS > 0), GpuEccDoubleBitError ( rate(ECC_DBE_VOL_TOTAL[10m]) > 0), GpuTemperatureHigh ( > 85°C for 10 m).

Throttle reasons: DCGM_FI_DEV_CLOCK_THROTTLE_REASONS (bitmap: bit3=HW thermal, bit4=HW power brake, bit6=HW slowdown), DCGM_FI_DEV_THERMAL_VIOLATION (cumulative µs), DCGM_FI_DEV_POWER_VIOLATION (cumulative µs). Verify which fields exist in your exporter ( curl ... | grep -iE 'throttle|violation') before writing rules. Example: rate(THERMAL_VIOLATION[5m]) > 0.05 for 10 m.

Threshold Derivation Methodology

Agree SLO with product (e.g., TTFT P95 ≤ 1.2 s, TPOT P95 ≤ 80 ms).

Load-test fixed input/output lengths (e.g., 1024/256) stepping up to design QPS; record TTFT percentiles at each step using vllm bench serve.

Set alert threshold = max(load-test P95 × 1.5–2, SLO × 1.15). Example: load-test P95=613 ms, SLO=1200 ms → threshold ≈ 1380 ms.

Two-tier rules per metric: > SLO×1.15 for 10m → warning (degradation), > SLO×3 for 2m → critical (likely unavailable). Single threshold causes alert fatigue or late detection.

Recording Rules to Protect Prometheus

Pre-aggregate histogram buckets: vllm:job:ttft_bucket_rate5m (

sum by (job, le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))

) — must keep le label or histogram_quantile breaks. vllm:job:tokens_per_sec ( sum by (job) (rate(vllm:generation_tokens_total[2m]))).

Grafana Dashboard Design

Four-Row Layout Following Troubleshooting Flow

SLO Overview : four numbers — current QPS, TTFT P95, TPOT P95, error/abort rate. Top row: "Is service healthy?"

Three-Layer Trend Correlation : shared time axis — request latency (top), engine queue/KV (middle), GPU UTIL/TENSOR_ACTIVE (bottom). Coincident inflection points reveal root cause instantly.

Throughput & Cost : input/output token rates, tokens/sec/GPU, prefix-cache hit rate. Capacity planning & cost optimization.

Hardware Health : temp, power, ECC, XID, VRAM trend. Ignored daily; lifesaver during incidents.

Key Panel PromQL

Tokens/sec/GPU :

sum by (job) (rate(vllm:generation_tokens_total[2m])) / count by (job) (DCGM_FI_DEV_GPU_UTIL)

. Align job labels or use label_join via node / pod.

Prefix-cache hit rate :

sum(rate(prefix_cache_hits[5m])) / clamp_min(sum(rate(prefix_cache_queries[5m])), 0.001)

. Counters are token-based; 40% hit = 40% of prompt tokens reused. RAG 50–70%; chat <10% normal.

Queue time share :

sum(rate(request_queue_time_seconds_sum[5m])) / clamp_min(sum(rate(request_queue_time_seconds_count[5m])), 0.001)

÷ avg E2E = queue contribution. >30% ⇒ capacity/scheduling issue, not model.

Input/output token ratio :

sum(rate(prompt_tokens_total[5m])) / clamp_min(sum(rate(generation_tokens_total[5m])), 0.001)

. >8:1 = long-input/short-output (RAG) → optimize prefill/prefix-cache. <0.5:1 = long-output → optimize KV cache/batch.

Misleading Indicators Annotated on Panels

GPU_UTIL 100%

≠ "compute saturated"; only "kernel running". DCGM_FI_DEV_FB_USED near total VRAM is normal (vLLM pre-allocates KV pool at gpu_memory_utilization), not a leak. num_requests_waiting = 0 could mean no traffic; always pair with QPS.

High finished_reason="length" ⇒ max_tokens or max-model-len too low; answers truncated.

Log-Metric Reconciliation

vLLM Stats Log (every 10 s via --log-stats-interval )

INFO Avg prompt throughput: 4123.5 tokens/s, Avg generation throughput: 1876.2 tokens/s,
Running: 42 reqs, Waiting: 3 reqs, GPU KV cache usage: 91.3%, Prefix cache hit rate: 38.7%

Cross-check with Prometheus: log KV 91.3% vs. panel kv_cache_usage_perc 0.63 ⇒ panel window or aggregation mismatch (e.g., max_over_time vs. instant). Query instant value in Prometheus UI first. Running: 42 vs. num_requests_running must match; systematic offset may indicate --api-server-count >1 splitting metrics/logging across processes — document known non-consistency.

Request-Level Tracing via Gateway

Enable --enable-request-id-headers (version-dependent) to propagate request IDs. Nginx log format captures uht (upstream header time ≈ TTFT for SSE) and urt (upstream response time ≈ E2E). Disable proxy_buffering for streaming. Verify token arrival is smooth (tens of ms gaps); bursts indicate buffering at CDN/LB layer.

Multi-Layer Reconciliation Example

Gateway: urt P99 = 11.2 s → Client truly slow

vLLM Request: e2e_request_latency_seconds P99 = 10.8 s → Slow inside vLLM

vLLM Request: time_to_first_token_seconds P99 = 6.4 s → Bulk in first token

vLLM Engine: request_queue_time_seconds mean = 5.9 s → Queue = 5.9 s

vLLM Engine: num_requests_waiting = 68 sustained → Queue backlog

vLLM Engine: kv_cache_usage_perc = 0.98 → KV full

vLLM Engine: num_preemptions_total rate = 3.1/s → Frequent preemption/recompute

GPU: DCGM_FI_DEV_GPU_UTIL = 100% → Surface full load

GPU: DCGM_FI_PROF_PIPE_TENSOR_ACTIVE = 0.07 → Compute actually idle

Root cause: KV exhaustion → preemption → prefill recompute → GPU busy but low effective compute → TTFT explosion. Fix: reduce max-model-len, max-num-seqs, tune prefix-cache, or shunt long-context requests to dedicated pool. Not simply "add GPUs".

Troubleshooting Decision Tree

User complains slow
│
├─ Request layer: TTFT P95 or TPOT P95 high?
│  ├─ TTFT high, TPOT normal
│  │  ├─ queue_time share >30% → queue issue
│  │  │  ├─ waiting long & KV <0.9 → arrival > capacity → scale/rate-limit
│  │  │  └─ waiting long & KV >0.95 → KV shortage → lower max-model-len/max-num-seqs
│  │  ├─ queue_time share <10% → prefill slow
│  │  │  ├─ prefill_time high & prompt tail long → long prompt → enable chunked prefill
│  │  │  └─ prefill high but prompt normal → TENSOR_ACTIVE low? check throttle, prefix-cache loss
│  │  └─ neither high but TTFT high → sampling/post-process overhead → check model_execute_time
│  ├─ TPOT high, TTFT normal
│  │  ├─ running small & TPOT high → batch too small, decode BW-limited (normal, hard to optimize)
│  │  ├─ running large & TPOT high → compute saturated, per-GPU token rate at ceiling → scale
│  │  ├─ running sawtooth & preemptions rising → preemption/recompute
│  │  └─ ITL P99 high but TPOT mean normal → tail jitter → check CPU steal, log flush
│  └─ TTFT/TPOT normal, E2E high
│     ├─ output_tokens tail long → normal, user asked for long answer
│     └─ SSE transport slow → gateway buffering, client BW, CDN
└─ GPU layer: XID/ECC/throttle/temp anomaly?
   ├─ Yes → fix hardware first (pollutes all upper metrics)
   └─ No → back to engine layer

Five Common On-Call Cases

GPU 100% util, but 300 tok/s/GPU (H100 60B should do 2000+) : Check TENSOR_ACTIVE (0.04), DRAM_ACTIVE (0.88), running=2 → tiny batch, decode memory-bound. Hidden cause: gateway keepalive/pool misconfig serializing requests; verify concurrent connections with ss -tn state established '( dport = :8000 )' | wc -l.

TTFT jumps 600 ms → 4 s, KV usage drops : Not capacity. KV drop = concurrency fell. Check request_prefill_time_seconds distribution, prefix_cache_hits/queries (step drop), recent config changes ( kubectl rollout history), CPU steal ( sar -u 1 5). Real cause: system prompt/format change moved prefix-cache hit point; hit rate 60% → 5%. Fix: move variable user info from system prompt to message tail.

P99 stuck at fixed value (e.g., 60 s), P50 normal : Fixed P99 = timeout, not real latency. Hunt timeout configs ( grep timeout /etc/nginx/conf.d/*.conf, deployment YAML). abort rate 15% confirms: requests cut at 60 s, vLLM records 60 s latency. Fix: align max_tokens and proxy_read_timeout; don't just raise timeout (worsens queue).

OOM restart, log shows CUDA out of memory : Not leak; concurrency/length exceeded pre-alloc. Check kv_cache_usage_perc hit 1.0 before OOM, request_prompt_tokens P99 vs. max-model-len/2. Typical: max-model-len > model limit, gpu-memory-utilization >0.95 no headroom for activation peaks, or multi-process + large batch. Conservative fix: --gpu-memory-utilization 0.88 + cap max-num-seqs.

One replica 3× slower, others fine : Environment/hardware diff. Per-replica metric pull + node GPU health ( nvidia-smi -q | grep -A3 'Link Width', dmesg -T | grep -i xid). Common: NVLink degraded or HBM ECC errors forcing downgrade. Won't self-heal; replace card.

Operational Risks & Production Hardening

Cardinality explosion : ~20 vLLM histograms × 15–30 buckets = 400+ series/replica. 30 replicas = 12k series + DCGM PROF. Mitigation: limit --custom-histogram-buckets ≤12 buckets, drop python_* / process_open_fds via metric_relabel_configs.

Cross-node scrape latency : Network blips → up==0 false alerts. scrape_timeout ≤60% of interval; inhibit up==0 with 2 m window before alerting.

Alert storms : Node down triggers up==0, VllmQueueBacklog, VllmHighTTFT, GpuXidError simultaneously. Inhibit rules: node-level critical suppresses all pod-level warnings on same instance.

Change Management

New/modified alert rules: promtool check rules + promtool test rules in staging, then 10% notification canary for 24 h.

Scrape interval changes alter TSDB growth (H100 node + vLLM ≈ 3k–6k series; 15 s → ~200–350 MB/day; 5 s triples disk).

Grafana dashboards as versioned JSON in Git; no live edits (lost on upgrade/migration).

Security

/metrics

unauthenticated. K8s NetworkPolicy allowing only monitoring and ingress-nginx namespaces to port 8000. Verify no other consumers (e.g., bench namespace) silently break.

Grafana: view-only for most; datasource read-only mode.

DCGM profiling metrics may include GPU serial numbers; strip from external reports.

Six Wall-Postable Rules

GPU utilization is an explanatory metric, not a health metric. Judge from request layer down; GPU metrics explain why request layer degraded. 100% util → ask "effective compute?" Use PROF_PIPE_TENSOR_ACTIVE / PROF_DRAM_ACTIVE to separate compute-bound vs. memory-bound.

Latency must be decomposed. TTFT = queue + prefill; E2E = TTFT + (N-1)×TPOT. Every "slow" decomposes to capacity, scheduling, or hardware.

Throughput must split input/output. Single "total tokens" hides workload shape. Input/output ratio dictates optimization: high ratio → prefix-cache/prefill; low ratio → KV cache/batch.

Verify metric names before use. vLLM/DCMG metric sets change across versions. Every dashboard/alert: curl /metrics → diff against actual list.

Thresholds from load tests, not copying. SLO → baseline → headroom → split warning/critical. Untested thresholds fail when needed most.

Hardware issues first. XID/ECC/throttle corrupt all upper metrics. Independent hardware-health layer with higher alert priority.

Key insight: inference monitoring differs from traditional web because resources are hard (shared GPUs/KV pool), time is long (seconds per request), and batching is shared (one long prompt drags everyone's TPOT). This interdependence demands three layers on one timeline — any single layer yields wrong conclusions.

Appendix: Reusable Scripts & Runbooks

One-Shot Snapshot Script ( vllm-snapshot.sh )

Read-only collection: full /metrics from vLLM + DCGM, key gauges ( num_requests_running/waiting, kv_cache_usage_perc), finished reasons, latency histogram sum/count, nvidia-smi CSV + compute apps. Outputs timestamped directory for postmortem. Usage:

./vllm-snapshot.sh <vllm_host:port> <dcgm_host:port> <outdir>

.

TTFT Degradation Runbook

Stepwise table: confirm real degradation (vs. 7d baseline) → check queue ( waiting >20) → check KV ( >0.95) → branch: KV shortage (check prompt token dist vs. max-model-len) or capacity (QPS vs. load-test knee) → check prefill time → check prefix-cache hit rate step-change → check hardware throttle/XID. Each step has concrete command/query, decision criterion, and explicit next action — no vague "observe".

Baseline Load-Test Script ( baseline.sh )

Stepped request-rate (2,4,8,12,16,24) with fixed input/output lengths, --num-prompts = rate × 30, 20 s cooldown between steps (drains residual KV). Output per-step JSON + log. Pitfalls: insufficient prompts ⇒ throughput inflated; no cooldown ⇒ residual KV contaminates next step; client/server co-location ⇒ CPU steal biases TTFT. Knee point where P95 TTFT spikes = capacity limit. Tokens/sec/GPU = fair cross-deployment metric.

Capacity Planning Integration

Example: load test yields 14 QPS/GPU at TTFT P95 ≤1.2 s (1024/256 mix). Business peak 40 QPS, N+1 redundancy. ceil(40/14)=3 min replicas; +1 spare = 4 replicas. Three assumptions: (1) per-GPU QPS valid only for tested config (model, max-model-len, TP); (2) 14 QPS at SLO — relaxing TTFT to 3 s might yield 30 QPS, halving cost; SLO and cost coupled. (3) Input/output lengths per test — RAG 8K+ prompts may drop to 2–3 QPS/GPU. Plan for peak ×1.3 burst headroom; prefer extra replica over raising max-num-seqs (amplifies blast radius on replica failure).

Common Objections & Rebuttals

"GPU util already 95%, why more monitoring?" → 95% could be healthy batch or KV-preemption spin; per-GPU tokens differ 5×. Without PROF_PIPE_TENSOR_ACTIVE + num_preemptions_total you can't tell.

"Prometheus too heavy; we use cloud vendor monitoring." → Cloud metrics usually only GPU util + VRAM; 80% of inference faults live in engine layer (queue, KV, prefix-cache). Must ingest vLLM /metrics at minimum.

"Too many alerts, on-call ignores them." → Alert design failure. Demote warnings to tickets/daily digest; only critical pages. Inhibit node-level faults. Every annotation must state "next diagnostic step" so on-call can act, not just escalate.

"Why not use vLLM's built-in dashboard?" → Use it, but verify metric names per 4.2 and re-derive thresholds per 7.4. Community dashboard saves layout work, not metric understanding.

"Can a single health score replace all these metrics?" → Composite scores hide actionable detail. Score 85/100 still doesn't tell you to lower max-model-len or scale out. Scores for management; engineering decisions need itemized metrics.

Go-Live Checklist

[ ] Versions recorded (vLLM, DCGM Exporter, Prometheus) and match running.

[ ] curl /metrics dumped; every dashboard/alert metric exists.

[ ] Histogram buckets tuned to SLO (or documented error bounds).

[ ] gpu_memory_utilization × max-model-len × max-num-seqs validated no OOM.

[ ] Every alert has annotation with diagnostic next step.

[ ] Alerts verified by "inject failure" test (load test, config tweak) — confirm firing.

[ ] Inhibit rules cover node-down storm.

[ ] /metrics ports restricted via NetworkPolicy/firewall.

[ ] Load-test baselines archived; threshold derivation documented.

[ ] Dashboard JSON in Git; no live edits.

[ ] Snapshot script validated in staging (paths, deps).

[ ] Runbook shared with on-call; at least one drill completed.

Once live, daily ops effort drops — most issues surface as "TTFT P95 creeping up" or "prefix-cache hit rate sliding" before becoming incidents. Monitoring's value isn't post-mortem charts; it's making anomalies visible before they become accidents.