Which Prometheus Metrics Actually Deserve a 2 AM Wake-Up Call?

Core Principle: Alerts Must Meet Five Criteria

An alert deserves a phone call only if it: (1) already causes user-visible impact or has strong evidence impact is imminent; (2) persists beyond a waiting window, not a transient spike; (3) gives the on-call engineer a concrete action that can reduce or contain impact; (4) can be verified with a single query, log check, or status command — no guessing; (5) has a bounded scope (single instance, AZ, endpoint, or full site).

Four-Tier Alert Classification

Tier            | Focus                                                      | Phone Alert?       | Typical Evidence
----------------|------------------------------------------------------------|--------------------|-------------------------------------------------
User Impact     | Availability, error rate, key endpoint latency             | Usually yes        | Black-box probes, business error rate, SLO
Service Saturation | Queues, concurrency, thread pools, connection pools, rate limits | When sustained threshold breached | Queue length, rejection count, latency percentiles
Resource Risk   | Memory reclaim, disk space, inodes, file descriptors       | Maybe              | Resource trend combined with business impact
Diagnostic Clues| CPU, NIC, single process, log volume                       | Usually no         | Host metrics and log context

CPU and disk metrics are candidate causes , not user-impact evidence. Design alerts so phone alerts land in the first two tiers; use the latter two for context and automated tickets.

Validate the Monitoring Pipeline First

1. Confirm Prometheus Scrapes Successfully

#!/usr/bin/env bash
set -euo pipefail
PROM_URL="<Prometheus地址>"
curl --fail-with-body --silent --show-error 
  "$PROM_URL/api/v1/targets" 
  | jq '.data.activeTargets[] | {scrapeUrl, health, lastError, lastScrape, scrapeInterval}'

Scrape failures are important monitoring-system alerts but must route separately from business unavailability; otherwise every exporter restart triggers the same page.

2. Labels Must Express Blast Radius

Retain env, service, instance, region in scrape configs. Never put high-cardinality values (request IDs, full URLs, user IDs) into labels. Label design determines whether alerts can group by service/region/instance and controls storage/query cost.

Priority 1: User Cannot Access the Service

1. up == 0 Is Not Business Unavailability, But Sustained Failure Should Escalate

up{job="application"} == 0

sum by (service, env, region) (up{job="application"})
/
count by (service, env, region) (up{job="application"})
< 0.5

For a single-instance service, two continuous minutes at zero warrants a page; for multi-instance services, use overall available-instance ratio and actual request failure rate. The ratio query requires stable label sets; dynamic scaling may need minimum-capacity-aware thresholds.

2. Error Rate Is Closer to User Experience Than CPU

Services should expose http_requests_total (or _total counters per OpenMetrics). Example 5xx rate with volume guard:

sum by (service, route, env) (
  rate(http_requests_total{
    service="<业务名>",
    status=~"5.."
  }[5m])
)
/
clamp_min(
  sum by (service, route, env) (
    rate(http_requests_total{service="<业务名>"}[5m])
  ),
  1
)
> 0.05

(
  sum by (service, route, env) (
    increase(http_requests_total{
      service="<业务名>",
      status=~"5.."
    }[10m])
  ) >= 20
)
and
(
  sum by (service, route, env) (
    rate(http_requests_total{
      service="<业务名>",
      status=~"5.."
    }[5m])
  )
  /
  clamp_min(sum by (service, route, env) (
    rate(http_requests_total{service="<业务名>"}[5m])
  ), 1)
  > 0.05
)

First expression: 5xx proportion over 5m. Second: requires ≥20 failures in 10m and 5xx rate >5%. clamp_min avoids near-zero denominator. Thresholds should derive from business capacity and SLO error budget.

3. Latency Must Use Percentiles, Not Averages

With Prometheus histograms ( http_request_duration_seconds_bucket, _sum, _count):

histogram_quantile(
  0.99,
  sum by (le, service, route, env) (
    rate(http_request_duration_seconds_bucket{
      service="<业务名>"
    }[5m])
  )
) > 1

(
  histogram_quantile(
    0.99,
    sum by (le, service, route, env) (
      rate(http_request_duration_seconds_bucket{
        service="<业务名>"
      }[5m])
    )
  ) > 1
)
and
(
  sum by (service, route, env) (
    rate(http_requests_total{service="<业务名>"}[5m])
  ) > 0.5
)

Threshold unit is seconds. Two queries: tail latency alone, and tail latency with sample-rate guard (>0.5 req/s). Low-traffic endpoints should combine black-box probes and business criticality.

Priority 2: Service Approaching Saturation

Resource utilization ≠ saturation. Watch for queuing, rejections, capacity limits.

1. Queue Depth and Concurrency Are the Most Valuable Saturation Signals

max by (service, queue, env) (
  app_queue_depth{service="<业务名>"}
)
> 1000

app_queue_depth{service="<业务名>"}
/
clamp_min(
  sum by (service, queue, env) (
    rate(app_queue_completed_total{service="<业务名>"}[5m])
  ),
  0.001
)
> 60

First query: absolute depth. Second: depth ÷ completion rate ≈ drain seconds. Fixed threshold 1000 is only an initial guard; zero consumption rate needs a separate consumer-stopped rule.

2. Connection Pool Exhaustion Often Precedes Host CPU Issues

(
  app_pool_in_use{service="<业务名>"}
  /
  clamp_min(app_pool_max{service="<业务名>"}, 1)
) > 0.9
and
app_pool_waiters{service="<业务名>"} > 0

High usage without waiters may be normal load; high usage with waiters means requests are blocked. Include pool name and instance labels to distinguish single-instance leak from global capacity shortage.

Host Metrics: Only Escalate When Tied to Risk or User Impact

1. CPU Alone Should Not Page

100 *
(1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
))
> 80

80% CPU ≠ failure. If success rate, latency, and queues are normal, just trend it. Escalate only when CPU high and business errors rise and run queue persistently exceeds CPU count:

(
  node_load1
  / on (instance)
  count by (instance) (
    node_cpu_seconds_total{mode="idle"}
  )
) > 2
and
(
  sum by (instance) (
    rate(http_requests_total{service="<业务名>", status=~"5.."}[5m])
  ) > 0
)

node_load1

counts runnable + uninterruptible tasks, not pure CPU. Thresholds must account for core count, workload type, and baseline.

2. Memory Alerts Should Use Available Memory and Reclaim Pressure

Linux uses free memory for cache; MemFree low is not proof of shortage. MemAvailable approximates allocatable memory.

(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.10

(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.05
and
(rate(node_vmstat_pgmajfault[5m]) > 10
  or rate(node_vmstat_pswpin[5m]) > 1
  or rate(node_vmstat_pswpout[5m]) > 1)

First query: warning. Second: critical — combines available memory <5% with major faults or swap activity. node_vmstat_* existence depends on enabled collectors.

3. Disk Space: Watch Time-to-Full, Not Just Percentage

(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay", mountpoint!~"/(proc|sys|dev|run)($|/)"}
/
node_filesystem_size_bytes{fstype!~"tmpfs|overlay", mountpoint!~"/(proc|sys|dev|run)($|/)"})
< 0.10

Excludes pseudo-filesystems; adjust exclusion list per environment. Alerts must include instance, mountpoint, device for quick targeting.

Linear prediction for time-to-full:

predict_linear(
  node_filesystem_avail_bytes{fstype!~"tmpfs|overlay", mountpoint!~"/(proc|sys|dev|run)($|/)"}[6h],
  24 * 3600
) < 0

predict_linear

assumes steady linear growth; log rotation, batch jobs, and temp files break it. Use for early-warning tickets, not as replacement for hard threshold protection.

Inode exhaustion:

(node_filesystem_files_free{fstype!~"tmpfs|overlay", mountpoint!~"/(proc|sys|dev|run)($|/)"}
/
node_filesystem_files{fstype!~"tmpfs|overlay", mountpoint!~"/(proc|sys|dev|run)($|/)"})
< 0.05

Small-file caches, mail queues, and temp directories are prone. Space and inode alerts should be separate; remediation differs.

4. Disk I/O: Escalate Only on Latency or Queueing

(rate(node_disk_read_time_seconds_total[5m])
+ rate(node_disk_write_time_seconds_total[5m]))
/
clamp_min(
  rate(node_disk_reads_completed_total[5m])
+ rate(node_disk_writes_completed_total[5m]),
  1
)
> 0.1

Approximates average device service time in seconds. It's a device-layer signal, not necessarily application thread wait. Correlate device name, mountpoint, and application latency to avoid false positives from high-throughput sequential I/O.

Monitoring System Self-Health Must Page

Prometheus process alive ≠ timely scrapes, evaluations, alert deliveries.

prometheus_rule_evaluation_failures_total > 0
or
prometheus_rule_group_last_duration_seconds
>
prometheus_rule_group_interval_seconds

Rule evaluation failures or duration exceeding interval delay real business alerts. Route to monitoring-platform admins, not shared business silencing rules.

(sum by (job) (rate(prometheus_target_scrapes_exceeded_sample_limit_total[5m])) > 0)
or
(sum by (job) (rate(prometheus_target_scrape_pool_exceeded_target_limit_total[5m])) > 0)

Sample/ target limits cause data loss, potentially understating error rates. Metric names depend on Prometheus version; verify via /metrics before deploying.

Package Rules into a Maintainable Alert Set

Example rule file (recording rules + critical alerts):

groups:
- name: service-slo
  interval: 30s
  rules:
  - record: service:http_5xx_rate5m
    expr: |
      sum by (service, route, env) (
        rate(http_requests_total{status=~"5.."}[5m])
      )
      /
      clamp_min(sum by (service, route, env) (
        rate(http_requests_total[5m])
      ), 1)
  - record: service:http_request_rate5m
    expr: |
      sum by (service, route, env) (
        rate(http_requests_total[5m])
      )
  - alert: ServiceHighErrorRate
    expr: |
      service:http_5xx_rate5m{service="<业务名>"} > 0.05
      and
      service:http_request_rate5m{service="<业务名>"} > 0.5
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "服务错误率持续升高"
      description: "service={{ $labels.service }}, route={{ $labels.route }}"
      runbook: "<故障手册地址>"
  - alert: ServiceTailLatencyHigh
    expr: |
      histogram_quantile(
        0.99,
        sum by (le, service, route, env) (
          rate(http_request_duration_seconds_bucket[5m])
        )
      ) > 1
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "服务 P99 延迟持续过高"
      description: "service={{ $labels.service }}, route={{ $labels.route }}"
      runbook: "<故障手册地址>"
  - alert: HostMemoryPressure
    expr: |
      node_memory_MemAvailable_bytes
      /
      node_memory_MemTotal_bytes < 0.05
      and
      rate(node_vmstat_pgmajfault[5m]) > 10
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "主机内存压力持续升高"
      description: "instance={{ $labels.instance }}"
      runbook: "<故障手册地址>"

Validate with promtool check rules <规则文件>, then reload via API and verify rule health:

#!/usr/bin/env bash
set -euo pipefail
PROMTOOL_BIN="<promtool路径>"
RULE_FILE="<规则文件>"
PROM_URL="<Prometheus地址>"
"$PROMTOOL_BIN" check rules "$RULE_FILE"
curl --fail-with-body --silent --show-error -X POST "$PROM_URL/-/reload"
sleep 2
curl --fail-with-body --silent --show-error "$PROM_URL/api/v1/rules" 
  | jq --arg file "$RULE_FILE" '
    .data.groups[]
    | select(.file == $file)
    | {name, rules: [.rules[] | {name, health, lastError}]}
  '

/-/reload

requires lifecycle API; otherwise use systemd or deployment system. sleep 2 is not a success guarantee.

Unit-Test Critical Rules

Complex error-rate and quantile rules need promtool test rules with sanitized real metric shapes:

rule_files:
- /tmp/service-alerts.yml

tests:
- interval: 1m
  input_series:
  - series: 'http_requests_total{service="checkout",route="/pay",status="200",env="prod"}'
    values: '100+100x10'
  - series: 'http_requests_total{service="checkout",route="/pay",status="500",env="prod"}'
    values: '0+20x10'
  alert_rule_test:
  - eval_time: 10m
    alertname: ServiceHighErrorRate
    exp_alerts:
    - exp_labels:
        service: checkout
        route: /pay
        env: prod
        severity: critical

Run promtool test rules <测试文件> in CI to catch syntax errors, missing labels, and logic regressions. Live API validation still needed for scrape targets, permissions, and remote config.

Alertmanager Routing Decides Who Gets Woken

global:
  resolve_timeout: 5m

route:
  receiver: ops-chat
  group_by: [alertname, service, env, region]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 2h
  routes:
  - matchers:
    - severity="critical"
    receiver: oncall-pager
    group_wait: 10s
    repeat_interval: 30m
  - matchers:
    - severity="warning"
    receiver: ops-chat

receivers:
- name: oncall-pager
  webhook_configs:
  - url: "<值班系统Webhook地址>"
    send_resolved: true
- name: ops-chat
  webhook_configs:
  - url: "<群聊Webhook地址>"
    send_resolved: true

group_by

must exclude high-cardinality labels or one incident splits into many notifications. repeat_interval too short causes fatigue; too long risks forgetting unresolved faults. Inject secrets via secret manager, not repo.

Validate config changes:

#!/usr/bin/env bash
set -euo pipefail
AMTOOL_BIN="<amtool路径>"
AM_CONFIG="<alertmanager配置文件>"
AM_URL="<Alertmanager地址>"
BACKUP_FILE="$AM_CONFIG.$(date +%Y%m%d%H%M%S).bak"
cp -- "$AM_CONFIG" "$BACKUP_FILE"
"$AMTOOL_BIN" check-config "$AM_CONFIG"
systemctl reload alertmanager
curl --fail-with-body --silent --show-error "$AM_URL/api/v1/status" 
  | jq '{cluster, versionInfo, config}'

If amtool check-config unsupported, use amtool --help or Alertmanager binary --config.file. On reload failure, restore backup; avoid repeated restarts that interrupt notifications.

During Alert Storms: Find Common Root Cause First

Don't process alerts in arrival order. Instead:

Are multiple services/instances/regions abnormal simultaneously?

Did all alerts start at the same time?

Is there an upstream network, DNS, cert, LB, or dependency failure?

Is it only scrape failures while user probes remain healthy?

Are alerts flapping (recover then re-fire)?

Pull current firing alerts via API:

#!/usr/bin/env bash
set -euo pipefail
PROM_URL="<Prometheus地址>"
curl --fail-with-body --silent --show-error 
  "$PROM_URL/api/v1/alerts" 
  | jq -r '
    .data.alerts[]
    | select(.state == "firing")
    | [.labels.severity, .labels.alertname, .labels.service, .labels.instance, .activeAt, .annotations.summary]
    | @tsv
  '

This shows firing alerts, not event timeline. For causality, examine activeAt, rule evaluation times, application logs, and change records.

Aggregate view of overall impact:

sum by (service, env) (
  rate(http_requests_total{status=~"5.."}[5m])
)
or
sum by (service, env) (
  rate(http_requests_total{status=~"4.."}[5m])
)

Whether 4xx counts as failure depends on business: client param errors usually shouldn't page like 5xx, but login failures or auth-service 4xx growth may indicate server-side issues. Split rules by status code, error type, and business semantics.

Silencing Is Not Fixing

Maintenance or acknowledged changes can silence alerts, but silences must have expiry, reason, and owner.

#!/usr/bin/env bash
set -euo pipefail
AMTOOL_BIN="<amtool路径>"
AM_URL="<Alertmanager地址>"
"$AMTOOL_BIN" 
  --alertmanager.url="$AM_URL" 
  silence add 
  --author="<值班人>" 
  --comment="<维护原因>" 
  --duration="1h" 
  alertname="<告警名>" service="<业务名>" env="<环境名>"

Broader matchers hide more risk. Never use alertname=~".*" or no-expiry silences as a substitute for repair. After adding, list silences and confirm only expected labels covered.

#!/usr/bin/env bash
set -euo pipefail
AMTOOL_BIN="<amtool路径>"
AM_URL="<Alertmanager地址>"
"$AMTOOL_BIN" 
  --alertmanager.url="$AM_URL" 
  silence query 
  --filter='alertname="<告警名>"' 
  --filter='service="<业务名>"'

After silence expires, verify alert recovers normally; notification resumption ≠ business stability.

Metrics That Mislead On-Call Engineers

Raw CPU, Memory, Disk Percentages

Necessary resource signals but lack workload semantics. CPU may be normal batch work; memory may be page cache; disk may be deletable logs. Phone alerts need duration + business impact + time-to-exhaust.

Single-Instance Process Restart Count

changes(process_start_time_seconds[15m]) > 0

detects restarts but with sufficient replicas, single-instance restart may not affect users. Keep as instance-level warning; escalate when service available-instance ratio drops.

Sudden Log Volume Increase

Log spikes are often diagnostic clues, not failures themselves. Repeated exception logs amplify disk risk, but logs don't prove request failures. Monitor structured error counters, log write errors, and disk time-to-full.

Build a Phone-Alert Allowlist

Instead of promoting rules from warning to critical, maintain an allowlist per alert recording: alert object, user impact, duration, first check command, escalation criteria, rollback action, recovery verification metrics.

Lightweight rule metadata checklist script:

#!/usr/bin/env bash
set -euo pipefail
RULE_FILE="<规则文件>"
grep -q '^ *- alert:' "$RULE_FILE"
grep -q '^ *for:' "$RULE_FILE"
grep -q '^ *severity:' "$RULE_FILE"
grep -q 'runbook:' "$RULE_FILE"
grep -q 'summary:' "$RULE_FILE"
grep -q 'description:' "$RULE_FILE"
echo "alert rule metadata checks passed"

This is a lightweight check, not a YAML parser. Production CI must also run promtool check rules and unit tests. Passing checks ≠ rule is live.

Root Cause Conclusions Must Rest on Evidence Chains

On-call goal: confirm impact, narrow scope, execute low-risk mitigation, preserve evidence. Record: user impact vs alert time, service scope, concurrent changes, key commands, pre/post mitigation error rate, latency, queue, rollback verification.

Snapshot inspection script:

#!/usr/bin/env bash
set -euo pipefail
PROM_URL="<Prometheus地址>"
SERVICE="<业务名>"
query() {
  local expression="$1"
  curl --fail-with-body --silent --show-error 
    --get "$PROM_URL/api/v1/query" 
    --data-urlencode "query=$expression" 
    | jq -c '.data.result'
}
query "sum(rate(http_requests_total{service=\"$SERVICE\",status=~\"5..\"}[5m]))"
query "sum(rate(http_requests_total{service=\"$SERVICE\"}[5m]))"
query "max(app_queue_depth{service=\"$SERVICE\"})"
query "count(up{job=\"application\",service=\"$SERVICE\"} == 1)"

Provides current snapshot; for before/after comparison use query_range or fixed time windows in dashboards.

Changes and Rollbacks Must Be Verifiable

Alerting system itself needs rollback. New rule causing storm → revert to previous version, retain failure samples, then analyze whether threshold, labels, aggregation, or routing was wrong. Don't delete rules — loses postmortem trail.

#!/usr/bin/env bash
set -euo pipefail
PROMTOOL_BIN="<promtool路径>"
PROM_URL="<Prometheus地址>"
RULE_FILE="<规则文件>"
BACKUP_FILE="<规则备份文件>"
cp -- "$BACKUP_FILE" "$RULE_FILE"
"$PROMTOOL_BIN" check rules "$RULE_FILE"
curl --fail-with-body --silent --show-error -X POST "$PROM_URL/-/reload"
curl --fail-with-body --silent --show-error "$PROM_URL/api/v1/rules" 
  | jq '.data.groups[] | .rules[] | {name, health, lastError}'

Confirm backup matches same environment and rule directory; ensure no other team's recent rules are overwritten. Reload success only means config read; still verify rule health, Alertmanager routing, and actual notifications.

Keep Only Signals That Form a Short, Reliable Path

After every 2 AM page, re-review: Did it map to real user impact? Was first step clear? Did the engineer need to guess across systems? If no, downgrade, add evidence, or delete.

A mature Prometheus stack doesn't aim to make every metric noisy; it makes phone alerts a short reliable chain:

User impact or clear capacity risk → duration threshold met → alert grouped by service/scope → notification includes first check command and runbook → engineer confirms root cause via metrics, logs, changes → executes reversible mitigation → verifies recovery via error rate, latency, queue, availability

More metrics demand clearer identification of the few that truly represent risk. Monitoring's value isn't creating more red; it's letting people know in the shortest time what's already hurting users, how much runway remains, and what's the safest action now.