PromQL Mastery: Node, Container, GPU & Business Metrics with Alerting Patterns
Comprehensive guide covering PromQL fundamentals, node/container/GPU/business metrics, alerting anti-patterns, and production-grade rule libraries with real troubleshooting cases.
1. PromQL Foundations: Understanding Evaluation Mechanics
Prometheus stores time series , not tables. A series is uniquely identified by metric name + label set; samples are (timestamp, float) pairs. Two critical facts: labels define series identity (high-cardinality labels like request IDs explode series count), and samples are instantaneous values — interval functions like rate compute over historical samples at query time.
1.1 Four Metric Types
Counter : Monotonically increasing cumulative. Examples: node_cpu_seconds_total, http_requests_total Gauge : Instantaneous value that can go up/down. Examples: node_memory_MemAvailable_bytes, up Histogram : Bucket counts + sum + count. Example: http_request_duration_seconds_bucket Summary : Client-side quantiles. Example: go_gc_duration_seconds{quantile="0.99"} Common mistake: applying rate to Gauge (yields meaningless "bytes/sec increase") or comparing Counter directly. Correct Gauge usage:
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100.
1.2 Interval Vector Window Coupling
rate(x[5m])looks back 5 minutes, requiring ≥2 samples. With scrape_interval=15s, window should be ≥4×scrape_interval (60s). Too small ( [30s]) causes gaps and jitter; too large ( [30m]) smooths spikes. [5m] is default sweet spot (20 samples).
1.3 Vector Matching Pitfalls
Binary ops ( a * b) require identical label sets (or on / ignoring). Mismatch yields empty result — no error . Example: node_filesystem_avail_bytes * node_filesystem_size_bytes fails if labels differ. Fix:
node_filesystem_avail_bytes * on (instance, mountpoint) group_left node_filesystem_size_bytes. Iron law: empty results don't error; every alert rule must be verified in /graph before deployment.
2. Node Layer: 16 Essential Questions from CPU to Disk
2.1 CPU Utilization: Three Algorithms
# Algorithm 1: 100 - idle (common but optimistic on hyperthreaded)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Algorithm 2: sum/sum busy ratio (recommended for alerts)
100 - (100 * sum by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) / sum by (instance) (rate(node_cpu_seconds_total[5m])))
# Algorithm 3: Top-N hottest cores (per-core)
topk(5, avg by (instance, cpu) (rate(node_cpu_seconds_total{mode="idle"}[5m])))Algorithm 2 uses sum/sum (aggregate then ratio) vs Algorithm 1's avg (ratio then average). On dynamic core counts (containers, hotplug), sum/sum stays accurate. Algorithm 3 returns lowest-idle cores; convert to utilization:
topk(5, 100 - avg by (instance, cpu) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100).
iowait/steal debate: capacity planning includes iowait; "process fighting CPU" excludes it. Separate views:
# Real compute load (exclude iowait, steal)
100 * (sum by (instance) (rate(node_cpu_seconds_total{mode=~"user|system|nice"}[5m])) / sum by (instance) (rate(node_cpu_seconds_total[5m])))
# IO wait share
100 * (sum by (instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m])) / sum by (instance) (rate(node_cpu_seconds_total[5m])))steal(hypervisor/container stolen time) must be monitored separately — unfixable by tenant.
2.2 Load & Saturation
# Load per core
node_load1 / count by (instance) (node_cpu_seconds_total{mode="idle"})
# Trend: 1m vs 5m load ratio
node_load1 / node_load5Interpretation: ratio <0.7 normal, 0.7-1.0 warning, >1.0 sustained 10min = queuing. node_load1/node_load5 > 1.5 signals rapid rise even if absolute load low.
2.3 Memory: Use MemAvailable, Not MemFree
# Correct: MemAvailable (excludes reclaimable PageCache)
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
# Wrong: MemFree (always 5-15% on healthy Linux)
100 * (1 - node_memory_MemFree_bytes / node_memory_MemTotal_bytes)MemAvailable= kernel estimate of memory for new processes without swap (includes reclaimable PageCache/slab). MemFree yields permanent 90%+ "usage" causing alert fatigue. Additional views:
# PageCache share
100 * node_memory_Cached_bytes / node_memory_MemTotal_bytes
# Swap used (absolute, not ratio)
node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes
# Kernel SLAB (dentry/inode leak early signal)
node_memory_Slab_bytes / 1024 / 10242.4 Disk: Three Separate Space Metrics
# FS usage (exclude tmpfs, overlay, docker layers)
100 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs", mountpoint!~"/var/lib/docker/.+"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"} * 100)
# Inode usage (small files exhaust this first)
100 - (node_filesystem_files_free{fstype!~"tmpfs|overlay"} / node_filesystem_files{fstype!~"tmpfs|overlay"} * 100)
# 7-day fill prediction (linear regression)
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 7*24*3600) < 0predict_linearextrapolates 6h slope to 7 days; negative = fills within 7 days. For disk growing 20%→40%, absolute threshold never fires but linear prediction alerts immediately.
2.5 Disk IO: Distinguish Busy vs Slow
# Device saturation (utilization)
rate(node_disk_io_time_seconds_total[5m]) * 100
# Avg request latency (ms) — business-facing
rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) * 1000
# Queue depth
rate(node_disk_io_time_weighted_seconds_total[5m])
# % time queue non-zero
rate(node_disk_io_time_seconds_total{device="sda"}[5m])Interpretation: SSD io_time >80% = saturated; await (avg latency) >5ms SSD / >20ms HDD needs action. High io_time + low await = busy but not slow (may not need action); low io_time + high await = real fault (single queue stall or remote storage timeout).
2.6 Network: Drop vs Retransmit
# NIC receive drop rate
rate(node_network_receive_drop_total[5m]) / rate(node_network_receive_packets_total[5m]) * 100
# TCP retransmit rate (kernel view, closer to app)
rate(node_netstat_Tcp_RetransSegs[5m]) / rate(node_netstat_Tcp_OutSegs[5m]) * 100
# Connection failures
rate(node_netstat_Tcp_AttemptFails[5m])Retransmit >0.5% investigate, >2% impacts business. node_netstat_* requires --collector.netstat (often disabled by default) — another empty-result trap .
2.7 Single Health Score for On-Call Triage
( (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 40
+ (node_load5 / count by (instance) (node_cpu_seconds_total{mode="idle"}) > 0.8) * 30
+ (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 30 )Not for alerting; for human dashboard: sort nodes by score to prioritize investigation.
3. Container & K8s Layer: cAdvisor vs kube-state-metrics
3.1 Two Data Sources, Different Semantics
cAdvisor ( container_*, machine_*): Runtime actual resource usage — How much CPU/memory used
kube-state-metrics ( kube_*): API object state & declared values — Desired replicas, readiness
Using cAdvisor for replica count or kube-state-metrics for memory pressure is wrong.
3.2 Container CPU: Filter PLEG & Pause Containers
sum by (namespace, pod, container) (
rate(container_cpu_usage_seconds_total{container!="", image!="", namespace="prod-core"}[5m])
) / sum by (namespace, pod, container) (
kube_pod_container_resource_limits{resource="cpu", namespace="prod-core"}
) * 100Three mandatory filters: container!="" (drop pod-level rollup), image!="" (drop POD sandbox/PLEG), namespace="prod-core" (never full-cluster scan — label filtering is first performance defense).
3.3 Container Memory: Working Set vs RSS
# Working set (excludes inactive_file) — matches kubelet OOM logic
sum by (namespace, pod) (container_memory_working_set_bytes{container!="", namespace="prod-core"})
# vs limit, >0.9 = OOMKilled precursor
sum by (namespace, pod) (container_memory_working_set_bytes{container!="", namespace="prod-core"}) / sum by (namespace, pod) (kube_pod_container_resource_limits{resource="memory", namespace="prod-core"})Why working_set not rss/usage? kubelet OOM uses working_set = usage - inactive_file (excludes reclaimable page cache). rss and usage overestimate; only working_set aligns with kubelet.
3.4 Restarts & Readiness: Separate Metrics
# Container restarts (kube-state-metrics)
increase(kube_pod_container_status_restarts_total{namespace="prod-core"}[1h]) > 0
# Ready vs desired replicas
kube_deployment_spec_replicas{namespace="prod-core"} - kube_deployment_status_replicas_available{namespace="prod-core"}
# Non-Running pods (phase label)
kube_pod_status_phase{namespace="prod-core", phase=~"Pending|Failed|Unknown"} == 1Critical: kube_pod_status_phase creates series per phase (Running=0/1, Pending=0/1, etc.). Cannot use phase!="Running" — a Pending pod has phase="Running"=0 and phase="Pending"=1. Must filter phase=~"Pending|Failed|Unknown" and == 1.
3.5 Request vs Actual: Capacity Governance Core
# requests vs actual ratio, long-term <20% = over-requested
sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[5m])) / sum by (namespace) (kube_pod_container_resource_requests{resource="cpu"})
# Long-term >90% = under-requested (scheduler overpacks)Requests drive scheduling; actual usage drives cost. Divergence = waste or instability.
3.6 Zombie/Idle Container Series
# Ended but still-referenced container series (cause orphaned memory points)
count by (namespace, pod) (container_memory_working_set_bytes{container!=""} > 0) != on (namespace, pod) group_left count by (namespace, pod) (kube_pod_info{namespace="prod-core"})Diagnoses "Pod deleted but curve persists" — cAdvisor series cleanup lag.
4. GPU Layer: dcgm-exporter Correct Usage
4.1 Metric Naming & Key Fields
Metrics prefixed DCGM_FI_; labels: gpu (index), device (e.g., nvidia0), modelName, Hostname.
DCGM_FI_DEV_GPU_UTIL : GPU compute core utilization (Gauge)
DCGM_FI_DEV_MEM_COPY_UTIL : Memory bandwidth utilization (Gauge)
DCGM_FI_DEV_FB_USED : Used framebuffer (MiB) (Gauge)
DCGM_FI_DEV_FB_FREE : Free framebuffer (MiB) (Gauge)
DCGM_FI_DEV_GPU_TEMP : Temperature (Gauge)
DCGM_FI_DEV_POWER_USAGE : Power draw (Gauge)
DCGM_FI_DEV_XID_ERRORS : Last XID error code (Gauge)
DCGM_FI_PROF_SM_ACTIVE : SM active ratio (0-1) (Gauge)
DCGM_FI_PROF_DRAM_ACTIVE : DRAM active ratio (0-1) (Gauge)
Note: DCGM_FI_PROF_* disabled by default; must enable in collector CSV.
4.2 GPU_UTIL Is Misleading
DCGM_FI_DEV_GPU_UTIL= "at least one kernel running" ratio. Two fatal flaws: (1) any thread running → 100% even if SM only 5% utilized; (2) doesn't distinguish compute vs memory-bound. Use PROF series:
# True compute utilization
avg by (gpu, Hostname) (DCGM_FI_PROF_SM_ACTIVE)
# Memory bandwidth utilization
avg by (gpu, Hostname) (DCGM_FI_PROF_DRAM_ACTIVE)
# Ratio: >1.5 = memory-bound, <1 = compute-bound
avg by (gpu) (DCGM_FI_PROF_DRAM_ACTIVE) / avg by (gpu) (DCGM_FI_PROF_SM_ACTIVE)LLM inference decode: DRAM_ACTIVE >0.7 + SM_ACTIVE <0.5 = normal decode profile (don't "optimize"). Prefill: high SM, low DRAM = normal.
4.3 Memory: Three Numbers Together
# Memory usage %
100 * DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)
# Fragmentation: memory grows but utilization flat
delta(DCGM_FI_DEV_FB_USED[30m]) > 2048 and on (gpu) delta(DCGM_FI_DEV_GPU_UTIL[30m]) <= 0
# ECC correctable errors (memory degradation signal)
increase(DCGM_FI_DEV_ECC_SBE_VOL_TOTAL[1h]) > 0Second query catches silent memory leak in inference: memory creeps up, utilization unchanged, eventually OOM. delta + and separates from noisy "high usage" alerts.
4.4 XID Errors: Only Hard Fault Signal
# Any XID error = immediate intervention
DCGM_FI_DEV_XID_ERRORS != 0XID 13/31 : Illegal memory access → App bounds check, inspect kernel code
XID 48 : Double-bit ECC error → Hardware fault, replace GPU
XID 63/64 : ECC page retirement → Log, plan replacement
XID 79 : GPU fallen off bus → Immediate node isolation
XID 94/95 : Container NVLink error → Check interconnect, may need reboot
XID 79 & 94 = immediate drain; others → ticket queue.
4.5 Temperature & Throttling: Hardware Issues Masquerading as Performance
# Temp >85°C warn, >90°C guaranteed throttle
DCGM_FI_DEV_GPU_TEMP > 85
# Power near limit
DCGM_FI_DEV_POWER_USAGE / DCGM_FI_DEV_POWER_MGMT_LIMIT > 0.95
# Throttle reason mask non-zero
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 0CLOCK_THROTTLE_REASONS != 0suffices; no need to decode bits. This alert catches "engine performance jitter" misdiagnoses: GPU throttles due to power wall → throughput drops 10-15% periodically while all software metrics look normal.
5. Business Layer: Four Golden Signals Done Right
5.1 Traffic: Use rate, Not increase
# QPS
sum by (service, status) (rate(http_requests_total{job="api-gateway"}[5m]))
# Error rate (5xx %)
sum by (service) (rate(http_requests_total{job="api-gateway", status=~"5.."}[5m])) / sum by (service) (rate(http_requests_total{job="api-gateway"}[5m])) * 100ratereturns per-second rate; increase returns window increment. For alerts, prefer rate: increase(x[5m]) > 100 ≡ rate(x[5m]) > 0.33, but latter decouples from window size — threshold unchanged when window changes.
5.2 Latency: histogram_quantile Required
# P99 (correct histogram method)
histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket{job="api-gateway"}[5m])))
# Average latency (distinct from quantile)
sum by (service) (rate(http_request_duration_seconds_sum[5m])) / sum by (service) (rate(http_request_duration_seconds_count[5m]))Three must-know limits: (1) sum by (le, ...) must keep le — dropping it yields NaN. (2) Quantile is bucket-boundary interpolation; default buckets (0.005-10s) too coarse for 100ms APIs. (3) Cannot average per-instance quantiles: avg(histogram_quantile(...)) wrong — must aggregate buckets first then compute quantile.
# Wrong: average of quantiles (underestimates tail)
avg by (service) (histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])))
# Correct: aggregate buckets then quantile
histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))5.3 Saturation: Queues & Thread Pools
# Thread pool saturation
thread_pool_active_threads / thread_pool_size
# Connection pool usage
db_connections_in_use / db_connections_max
# Queue growth rate (trend > absolute)
deriv(queue_depth[10m]) > 0deriv= per-second change; > 0 means queue growing. Trend-based alert far more valuable than static "queue > 100".
5.4 Errors: Rate vs Absolute Count
# Error rate (for SLO)
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
# Absolute errors (low-traffic fallback)
sum(increase(http_requests_total{status=~"5.."}[5m])) > 10Low-traffic services (<10 req/min) — error rate spikes wildly (one error = 10%). Must use absolute count or protect denominator:
sum(rate(http_requests_total{status=~"5.."}[5m])) / (sum(rate(http_requests_total[5m])) > 0.1 or vector(0.1))or vector(0.1)floors denominator at 0.1, avoiding 100% phantom at near-zero traffic.
6. Deep Water: Five Alert Failure Loops (Full Root-Cause Walkthroughs)
6.1 Loop 1: Disk Alert Silent 90 Days, Disk Filled
Phenomenon: /data partition fills at night, app crashes. NodeDiskFull rule never fired in 90 days.
Step 1 — Verify data exists: Query node_filesystem_avail_bytes{instance='10.0.1.31:9100'} → /data shows 0 bytes free. Data present.
Step 2 — Extract rule:
(node_filesystem_avail_bytes{mountpoint="/data"} / node_filesystem_size_bytes) * 100 < 10Step 3 — Root cause: Two errors: (a) numerator filters mountpoint="/data", denominator doesn't → vector mismatch → empty result (no error). (b) No fstype filter → tmpfs/overlay pollutes denominator.
Step 4 — Fix & verify:
100 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs", mountpoint="/data"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs", mountpoint="/data"} * 100) > 90Pre-fix: empty result array. Post-fix: returns 10.0.1.31 /data 100. This is why "must see real curve in /graph before deploy" (Section 1.4) matters — paid real price here.
Query returns series count : Before 0, After 1
Alert fires : Before No (silent failure), After Yes
Discovery time : Before 90 days (until failure), After Immediate
6.2 Loop 2: P99 Looks Better, Business Slower
Phenomenon: Post-deploy, Grafana P99 drops 820ms→310ms, but complaints rise. QPS/error rate unchanged.
Step 1 — Check if data volume dropped:
sum by (service) (rate(http_request_duration_seconds_count[5m]))→ same ~1240/sec. Not traffic drop.
Step 2 — Inspect histogram buckets: le=5 = 1188, +Inf = 1240 → 52 requests >5s exist, yet P99 shows 310ms.
Step 3 — Root cause: Middleware added 3s circuit breaker; timed-out requests moved to http_request_aborted_total, removed from latency histogram. Long tail vanished → P99 artificially improved.
Step 4 — Fix: Include aborted in error rate; annotate dashboard:
# True success rate: success / (success + fail + aborted)
sum(rate(http_requests_total{status!~"5.."}[5m])) / (sum(rate(http_requests_total[5m])) + sum(rate(http_request_aborted_total[5m])))Dashboard annotation mandatory: "Post 3s breaker, P99 excludes aborted requests — view with aborted_ratio."
P99 latency : Pre-change 820ms, Post-change (panel) 310ms, Post-change (true) 874ms
Abort ratio : Pre-change 0, Post-change (panel) 4.2%, Post-change (true) 4.2%
User-perceived success : Pre-change 99.97%, Post-change (panel) 95.8%, Post-change (true) 95.8%
Conclusion: Real P99 rose 820→874; 564ms "improvement" was statistical sleight-of-hand. Any observation scope change must update panel annotations AND alert baselines simultaneously.
6.3 Loop 3: Container Memory Alerts 200×/Day, All False
Phenomenon: ContainerMemoryHigh uses container_memory_usage_bytes / limit, threshold 0.85, fires 200+/day, zero real OOMKilled.
Step 1 — Compare alerting pods vs limits: Top pods at 91%, 88%, 86% of limit.
Step 2 — Compare usage vs working_set: For same pod: usage=1832MiB, working_set=1204MiB, limit=2048MiB. By kubelet OOM logic (working_set), actual pressure = 59% — nowhere near danger.
Step 3 — Root cause: usage_bytes includes page cache + kernel slab; kubelet uses working_set ( usage - inactive_file). This service reads logs heavily → page cache inflates usage to 91%, but cache is reclaimable, never causes OOM.
Step 4 — Fix:
- alert: ContainerMemoryHigh
expr: |
sum by (namespace, pod, container) (
container_memory_working_set_bytes{container!=""}
) / sum by (namespace, pod, container) (
kube_pod_container_resource_limits{resource="memory"}
) > 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.pod }} working set exceeds limit 90%"Added for: 10m — transient working_set spikes common; sustained 10min = real OOM risk.
Daily alerts : Before 213, After 4
Actual OOMKilled : Before 0, After 0
Missed real OOM : Before Yes (drowned in noise), After No
Second win > noise reduction: When alerts fire 200×/day, real OOM gets ignored too.
6.4 Loop 4: GPU 100% Utilization, 40% Throughput
Phenomenon: After scale-out, DCGM_FI_DEV_GPU_UTIL = 100%, throughput only 40% baseline.
Step 1 — Check PROF_SM_ACTIVE: 18-20% vs GPU_UTIL 100%. Confirms Section 4.2: GPU_UTIL=100% with any kernel running; one polling thread saturates it.
Step 2 — Check DRAM_ACTIVE: 20% — neither compute nor memory bound. GPU doing nothing?
Step 3 — Power & NVLink: nvidia-smi dmon shows 140W (limit 700W), SM 18-20%, MEM 19-21%, ENC/DEC 0%. GPU idle waiting for CPU .
Step 4 — CPU side: pidstat shows process CPU 50.6% (not saturated). vmstat: cs (context switches) 30-40k/sec, r (run queue) 9-12. Root cause: CPU oversubscription — multiple inference scheduler threads + host workloads → run queue > cores → GPU kernel launches delayed by CPU scheduling latency.
Step 5 — Fix: Pin inference process to dedicated physical cores (not hyperthreads):
# Check affinity
taskset -cp $(pgrep -f vllm | head -1) # shows 0-127 (all logical cores)
# Bind to physical cores 0-15 + hyperthreads 64-79
taskset -cp 0-15,64-79 21884
# Limit neighbor CPU quota
systemctl set-property app-worker.service CPUQuota=200%PROF_SM_ACTIVE : Before 18%, After 61%
Output throughput : Before 40% baseline, After 96% baseline
Context switches/sec : Before 32k, After 8k
Run queue (r) : Before 9-12, After 1-2
Core discriminant: DEV_GPU_UTIL=100% but PROF_SM_ACTIVE<30% → almost certainly CPU-side kernel starvation. Check pinning & context switches first; don't tune engine params.
6.5 Loop 5: histogram_quantile Returns NaN, Panel Blank
Phenomenon: P99 panel blank for report-api, but QPS panel works.
Step 1 — Query buckets directly:
sum by (le) (rate(http_request_duration_seconds_bucket{service="report-api"}[5m]))returns series with le — data source fine.
Step 2 — Inspect rule:
histogram_quantile(0.99, sum by (service) (rate(http_request_duration_seconds_bucket{service="report-api"}[5m])))— sum by (service) drops le → histogram_quantile receives bucket values without boundaries → NaN.
Step 3 — Fix inner aggregation: sum by (le) (rate(...)) keeps le. But panel shows NaN per le + duplicate curves.
Step 4 — Third trap: histogram_quantile output retains le label. Must strip explicitly:
histogram_quantile(0.99, sum without (instance) (rate(http_request_duration_seconds_bucket{service="report-api"}[5m])))Yields single clean series {} 2.418 (only service label). Rule: inner aggregation keeps le ; outer output must drop le .
Step 5 — Harden in CI:
grep -n 'histogram_quantile' /etc/prometheus/rules/*.yml | grep -v 'le' && echo "Possible missing le in histogram_quantile, review" || echo "No anomaly"Crude (false positives on without), but catches most pre-deploy.
7. Alert Rule Library: 16 Rules Across Four Layers
7.1 Node Layer (5 rules)
NodeCpuSaturated: compute load (user|system|nice) >85% for 10m NodeMemoryPressure: MemAvailable/MemTotal <10% for 5m NodeDiskWillFill: predict_linear(avail[6h], 4d) < 0 for 30m NodeDiskIoLatencyHigh: read await >20ms for 10m NodeTcpRetransHigh: TCP retransmit rate >2% for 10m
7.2 Container & GPU Layer (7 rules)
PodRestartLoop: restarts >3 in 30m (critical) PodNotReady: desired - available >0 for 10m (critical) ContainerMemoryWorkingSetHigh: working_set/limit >0.9 for 10m GpuXidError: XID != 0 for 1m (critical) GpuThrottled: throttle reasons != 0 for 5m GpuMemoryLeakSuspect: FB_USED delta >2GiB & GPU_UTIL delta <=0 for 30m GpuSmLowOnHighUtil: GPU_UTIL >90 & PROF_SM_ACTIVE <0.3 for 15m
7.3 Business Layer (4 rules)
HighErrorRate: 5xx rate >1% for 5m (critical) HighLatencyP99: histogram_quantile(0.99) >1s for 10m SloBurnFast: 1h error budget burn rate >14.4× (5% monthly budget in 2 days) for 2m (critical) LowTrafficAbsoluteErrors: 5xx count >10 in 5m (warning) SloBurnFast 's 14.4 comes from multi-window burn-rate standard: 1h window at 14.4× burn = 5% monthly budget exhausted in 2 days. Not arbitrary — derive before changing.
7.4 Three Pre-Deploy Checks
promtool check rules— syntax only promtool test rules — semantic test with synthetic series (only way to catch empty-result bugs pre-deploy)
Grep scan for Prometheus 3.x breaking patterns (see Section 8.4)
Example test file for NodeDiskWillFill:
rule_files: ["../node-layer.yml"]
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
- series: 'node_filesystem_size_bytes{fstype="xfs",mountpoint="/data",instance="n1"}'
values: '100 100 100 100 100 100'
- series: 'node_filesystem_avail_bytes{fstype="xfs",mountpoint="/data",instance="n1"}'
values: '100 100 100 12 12 12'
alert_rule_test:
- eval_time: 6m
alertname: NodeDiskWillFill
exp_alerts:
- exp_labels:
severity: warning
fstype: xfs
mountpoint: /data
instance: n1Uses "drop then plateau" (not drop to 0) because predict_linear needs ≥2 points for slope; constant values → slope 0 → no alert. Falling edge tests both slope fit and threshold.
Intentional typo mountpoint="/date" → test fails with got: [] (empty result) — this is the real deploy gate, not syntax check.
8. Performance: Prevent Queries from Becoming Incidents
8.1 Three Cost Indicators
Query /api/v1/status/tsdb:
{"series": [{"name":"container_cpu_usage_seconds_total","value":184233}, {"name":"container_memory_working_set_bytes","value":182917}, {"name":"node_cpu_seconds_total","value":46080}, {"name":"http_request_duration_seconds_bucket","value":31440}], "labels": [{"name":"pod_uid","value":91204}, {"name":"container_id","value":60118}, {"name":"pod","value":24038}]}container_*~180k series each (high cardinality); pod_uid 91k values, container_id 60k — classic high-cardinality labels. Any unfiltered container query scans 180k series — why Section 3.2 mandates namespace/job filters.
# Anti-pattern: full cluster scan
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m]))
# Good: narrow first
sum by (namespace) (rate(container_cpu_usage_seconds_total{namespace=~"prod-core|prod-data", container!=""}[5m]))8.2 Four Timeout Causes
Logs show two distinct failures: Query exceeded maximum time (reduce window/add filters) vs query max samples exceeded (single eval samples > --query.max-samples — reduce range or series).
global:
scrape_interval: 15s
evaluation_interval: 30s
query_range:
max_samples: 50000000
max_points_per_series: 11000max_points_per_seriesprotects long-range queries: 30 days at 15s = 172k points — browser can't render. Forces downsampling to 11k points; tradeoff: spikes invisible in long views.
8.3 Recording Rules: Heavy → Light
Precompute repeated complex expressions:
groups:
- name: rec-container
interval: 30s
rules:
- record: namespace:container_cpu_usage:rate5m
expr: |
sum by (namespace) (
rate(container_cpu_usage_seconds_total{container!=""}[5m])
)
- record: service:http_request_duration_seconds:p99
expr: |
histogram_quantile(0.99,
sum without (instance) (
rate(http_request_duration_seconds_bucket[5m])
)
)Dashboard uses namespace:container_cpu_usage:rate5m — cost drops from "scan 180k series" to "read one precomputed series". Naming convention: level:metric:operation (e.g., service:http_request_duration_seconds:p99) reveals aggregation level & operation. Cost: each recording rule = periodic eval; keep <500 per instance, else shard or migrate engine.
8.4 Prometheus 3.x Silent Behavior Changes (v3.16.0 stable Nov 2026)
v3.15.0+: Duration expressions in range selector offset error: foo[5m] offset (1m * 2) and foo[5m] @ start() now parse error (previously worked). Scan rules for these patterns.
v3.16.0+: Subquery samples count toward TotalSamples (hits max-samples sooner); type and unit labels dropped in binary ops — any query grouping/joining on them returns empty.
# Pre-upgrade scan
grep -nE 'offset *\(' /etc/prometheus/rules/*.yml || echo "No duration expr offset"
grep -nE '@ *(start|end)\(\)' /etc/prometheus/rules/*.yml || echo "No @ start/end"
grep -nE 'by *\(.*(type|unit)' /etc/prometheus/rules/*.yml || echo "No type/unit grouping"All three must return "No" before upgrade. These "syntax-legal but semantic-shift" changes evade promtool — only manual grep catches them.
9. Automation: Weekly Alert Rule Health Check Script
Rules accumulate two debts: dead rules (never fire) and flapping rules. Manual audit = half day; script runs weekly.
#!/usr/bin/env bash
# /usr/local/bin/promql_rule_doctor.sh
# Usage: promql_rule_doctor.sh <prometheus_url> [--dry-run]
set -euo pipefail
IFS=$'
\t'
PROM_URL="${1:?usage: promql_rule_doctor.sh <url> [--dry-run]}"
DRY_RUN="no"
[[ "${2:-}" == "--dry-run" ]] && DRY_RUN="yes"
LOOKBACK="${LOOKBACK:-7d}"
OUT="${OUT:-/var/log/promql_rule_doctor.log}"
need() { command -v "$1" >/dev/null 2>&1 || { echo "Missing dep: $1" >&2; exit 3; } }
need curl; need jq
echo "==== PromQL Rule Health Check $(date '+%F %T') ==="
rules_json=$(curl -sf --max-time 20 "${PROM_URL}/api/v1/rules" | jq '.data.groups[].rules |= map(select(.type=="alerting"))')
total=$(jq '[.data.groups[].rules[]] | length' <<< "${rules_json}")
echo "Total alert rules: ${total}"
echo "---- 1. Rules failing to load ----"
bad_health=$(jq -r '.data.groups[] as $g | $g.rules[] | select(.health != "ok") | "\($g.name)\t\(.name)\t\(.health)\t\(.lastError // "-")"' <<< "${rules_json}")
if [[ -z "${bad_health}" ]]; then echo " [ OK ] All rules loaded"; else echo "${bad_health}" | sed 's/^/ [FAIL] /'; fi
echo "--- 2. Rules never firing in ${LOOKBACK} ---"
while IFS=$'\t' read -r name; do
[[ -z "${name}" ]] && continue
resp=$(curl -sf --max-time 15 -G "${PROM_URL}/api/v1/query" --data-urlencode "query=count_over_time(ALERTS{alertname='${name}',alertstate='firing'}[${LOOKBACK}])" 2>/dev/null || echo '{"status":"error"}')
st=$(jq -r '.status' <<< "${resp}")
[[ "${st}" != "success" ]] && { printf ' [SKIP] %s
' "${name}"; continue; }
hits=$(jq -r '[.data.result[].value[1] | tonumber] | add // 0' <<< "${resp}")
printf ' %-42s trigger samples %s
' "${name}" "${hits}"
done < <(jq -r '.data.groups[].rules[].name' <<< "${rules_json}")
echo "---- 3. High-cardinality labels (value count >20k) ----"
tsdb=$(curl -sf --max-time 20 "${PROM_URL}/api/v1/status/tsdb")
jq -r '.data.labelValueCountByLabelName[] | select(.value > 20000) | " [WARN] \(.name) has \(.value) values, avoid grouping"' <<< "${tsdb}"
echo "---- 4. Top 5 series-heaviest metrics ----"
jq -r '.data.seriesCountByMetricName[0:5][] | " \(.value)\t\(.name)"' <<< "${tsdb}"
echo "---- 5. Scrape targets down ----"
targets=$(curl -sf --max-time 15 "${PROM_URL}/api/v1/targets")
down=$(jq -r '[.data.activeTargets[] | select(.health != "up")] | length' <<< "${targets}")
if [[ "${down}" -gt 0 ]]; then jq -r '.data.activeTargets[] | select(.health != "up") | " [FAIL] \(.labels.job) \(.lastError)"' <<< "${targets}"; else echo " [ OK ] All targets healthy"; fi
echo "==== Check complete $(date '+%F %T') ==="
[[ "${DRY_RUN}" == "yes" ]] && echo "(dry-run: report only, no changes)"Sample output highlights: NodeDiskWillFill & NodeTcpRetransHigh 0 hits in 7d — former needs expression audit (Loop 1), latter likely netstat collector disabled. High-cardinality scan flags pod_uid (91k) & container_id (60k) — avoid grouping by these. Safety design: set -euo pipefail, quoted vars, read-only GET, --dry-run explicit, missing dep exits with code 3.
10. Production Change Risk Matrix & Rollback
10.1 Four Change Classes
Alert rule edit : Impact scope = Notification & on-call chain. Rollback = git checkout then /-/reload Recording rule add : Impact scope = Query perf & rule eval latency. Rollback = Delete rule segment, reload
Global max_samples adjust : Impact scope = All dashboards & long-range queries. Rollback = Revert value, restart (not hot-reload)
Prometheus major upgrade : Impact scope = All query semantics, TSDB format. Rollback = Binary rollback + old data dir
Third row critical: query_range.max_samples not hot-reloadable — requires systemctl restart prometheus (WAL replay = tens of seconds to minutes downtime). Must run in maintenance window with advance notice to all dashboard consumers.
10.2 Reload vs Restart Boundary
# Safe: hot-reload rules & scrape config, no downtime, no TSDB clear
curl -s -X POST http://10.40.9.10:9090/-/reload
# Dangerous: process restart, WAL replay = query unavailable
systemctl restart prometheusConfig items supporting /-/reload: rule_files, scrape_configs, alerting. Require restart: global, storage.tsdb.*, query_range.*. CLI flags (e.g., --storage.tsdb.retention.time) need systemd unit edit + restart.
# Capture flags before/after as evidence
curl -s http://10.40.9.10:9090/api/v1/status/flags | jq '{retention: .data["storage.tsdb.retention.time"], max_samples: .data["query.max-samples"]}'10.3 Six-Element Change Checklist
Blast radius: Which alerts/panels/teams affected?
Pre-flight: promtool check rules PASS, promtool test rules PASS, three grep scans clean.
Backup: Rules dir in Git (tag pre-change); prometheus.yml & unit file to /etc/prometheus/bak/.
Window: Reload anytime; restart in low-traffic with notice.
Canary: New rule at severity: info for 1 week, verify hit rate & targets, then promote to warning.
Rollback command: git checkout HEAD~1 rules/ && curl -X POST .../-/reload — single executable line.
Canary most skipped, highest value for alerts: critical straight to prod wakes on-call on false positive; info for a week lets you silently observe hit count & targets.
11. Closing: Seven PromQL Engineering Iron Laws
Empty result ≠ error. Every rule must show real curve in /graph or have promtool test rules assertion.
Gauge ≠ rate; Counter ≠ direct threshold. Wrong type = meaningless result, not just "inaccurate".
histogram_quantile must keep le in inner aggregation, drop le in output. Miss one → blank panel or NaN storm.
Quantile aggregation order matters. sum by (le) buckets first, then histogram_quantile; reverse = tail erased.
DEV_GPU_UTIL ≠ GPU utilization. Real compute = PROF_SM_ACTIVE; memory-bound = PROF_DRAM_ACTIVE; both low → check CPU pinning.
Every query needs namespace/job/instance filter. 180k-series full scan becomes Prometheus self-inflicted incident on dashboard refresh.
Observation scope changes = code changes. Added circuit breaker? Changed sampling? Altered buckets? Update panel annotations AND reset alert baselines — or waste hours in "metrics improve, business degrades".
Each law maps to a real incident averaging >4h to resolve. PromQL syntax learnable in a day; the hard part is knowing which numbers to distrust .
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
