Building Enterprise Prometheus Monitoring: Unified Host, Container & Business Metrics Collection

1. Problem Background: Cost of Three Siloed Monitoring Systems

Most companies' monitoring grows organically: ops uses Zabbix for hosts, developers embed logs, containers add cAdvisor+Grafana, microservices expose /metrics. When failures occur, fragmented observability causes long diagnosis times. A typical incident: order service latency spikes; Zabbix shows host CPU 35%, container panel misses the namespace, logs reveal downstream connection pool exhaustion. Root cause: disconnected signals — host metrics in Zabbix (1m granularity, no container labels), container metrics in cAdvisor (7d retention, broken series on pod restart), business metrics via manual log grep, misaligned timestamps across systems.

Unified collection must solve five requirements: label correlation (host: instance/job; container: namespace/pod/container; business: service/endpoint), time precision consistency (same clock source, aligned scrape intervals), controllable retention (raw 15-30d, aggregated 1y), alert convergence (suppress duplicate alerts from same root cause), extensibility (new targets via config only). Prometheus fits with pull model, config-as-code, multi-dimensional labels, mature exporters. Limitations: high cardinality, single-node storage limits, long-term storage needs extra components.

2. System Overview: Component Responsibilities

Prometheus Server : Scrape, store, PromQL, rule evaluation. Not responsible for long-term storage, alert notification. Default port 9090.

Exporter : Translate target metrics to text format. Not responsible for aggregation, alerting. Various ports.

Pushgateway : Receive short-lived job pushes. Not responsible for replacing pull model, long-term cache. Default port 9091.

Alertmanager : Group, inhibit, silence, route, notify. Not responsible for rule evaluation, metric storage. Default port 9093.

Grafana : Visualization, dashboards, optional alerting. Not responsible for data collection, long-term storage. Default port 3000.

kube-state-metrics : Convert K8s object state to metrics. Not responsible for container resource usage. Default port 8080.

cAdvisor/kubelet : Container CPU, memory, network, filesystem. Not responsible for business semantics. Default ports 8080/10250.

Data flow: node_exporter, cAdvisor, app /metrics, kube-state-metrics → Prometheus (pull, 15s) → local TSDB (15d) → remote write → long-term storage; Prometheus rule evaluation → Alertmanager → DingTalk/Email/Webhook; Grafana queries Prometheus for dashboards.

Key boundaries: Don't use Grafana alerts instead of Alertmanager (weak grouping/inhibition/silencing, state in Grafana DB). Pushgateway only for CronJob/batch jobs (stale data if used for long-lived services). kube-state-metrics provides object metadata, not resource usage (use cAdvisor for CPU/memory).

3. Environment Assumptions & Version Conventions

OS: Rocky Linux 9.4 / Ubuntu 22.04 LTS (systemd, check SELinux)

Prometheus: 2.53.x LTS (long-term support, production preferred)

node_exporter: 1.8.x (1.9+ changes default collectors)

Alertmanager: 0.27.x (config compatible with 0.26)

Grafana: 11.x (dashboard JSON compatible with 10.x)

Container Runtime: containerd 1.7 / Docker 24.x (cAdvisor metric names consistent)

Kubernetes: 1.28–1.30 (kubelet includes cAdvisor)

Deployment: Binary+systemd (Prometheus/Alertmanager), containers (workloads) — easier troubleshooting/upgrades

Prometheus 3.x notes: UTF-8 metric names default, remote write 2.0 optional, some CLI flags removed. Configs work on 3.x but startup flags noted. Stay on 2.53 LTS unless strong need. Pre-upgrade: promtool check config and promtool check rules, read migration guide. Two silent-breaking changes: PromQL =~ / !~ become fully anchored (2.x {job=~"api"} matches api-server, 3.x only api — must write {job=~"api.*"}); holt_winters() renamed to double_exponential_smoothing(). These cause blank panels/silent alert failures.

Hardware baseline: Prometheus single node 8C/32GB/500GB SSD (NVMe preferred), dedicated /data mount; Alertmanager 2C/4GB two-node cluster; Grafana 2C/4GB with PostgreSQL backend; Scale: 200 physical/virtual machines, ~2000 containers, 30 business services, active series ≤1.5M. Beyond 10x scale, need sharding or VictoriaMetrics/Thanos.

4. Capacity Planning: Calculate Before Provisioning

4.1 Sample Rate Estimation

Formula:

samples_per_second = Σ (targets_i × series_per_target_i / scrape_interval_i)

. Per-target series estimates: node_exporter 800-1200 (default collectors), cAdvisor per container 200-400, kube-state-metrics 5k-30k (scales with Pod/Service/Endpoint count), business app (Go/Java) 100-2000. For baseline scale: hosts 200×1000/30s=6,667; containers 2000×300/30s=20,000; kube-state 1×15000/30s=500; business 30×500/15s=1,000. Total ~28,000 samples/sec. Business uses 15s for latency analysis; hosts 30s sufficient. Alert for duration must match coarsest granularity to avoid premature firing.

4.2 Storage Estimation

Official compressed sample cost ~1-2 bytes; use 1.5 bytes × 1.3 index/WAL overhead. Daily samples = 28,000 × 86,400 ≈ 2.42e9. Daily raw = 3.63 GB. 15d retention = 54.5 GB. With overhead ≈71 GB. Add 30% compaction headroom, 5-10 GB remote write WAL, recording rule series. 500 GB disk safe for 15d+30% headroom, can stretch to 25d. Minimum 200 GB.

4.3 Memory Estimation

Head block (last 2h) + query structures. ~2.5 KB per active series. 1.5M series × 2.5 KB ≈ 3.75 GB. Plus Go runtime, query cache, rule eval ≈ 8-12 GB. 32 GB provides headroom for large-range queries (30d in Grafana). If only 16 GB, limit query range and set max_source_resolution and query timeout.

4.4 Cardinality Baseline

Host: 200k series; key labels: instance, job, device, mountpoint; growth: new machines, mountpoints

Container: 600k series; key labels: namespace, pod, container, image; growth: pod recreation, transient Jobs

K8s Objects: 15k series; key labels: namespace, pod, resource; growth: object count growth

Business: 15k series; key labels: service, method, status, endpoint; growth: new endpoints, status codes

Middleware: 50k series; key labels: instance, db, queue; growth: new instances, DB/table labels

If actual exceeds 2x estimate, investigate before scaling. Typical cardinality explosion: user IDs, request IDs, URL paths as labels — adding memory won't fix.

5. Pre-flight Checks: 10 Must-Verify Items

5.1 Time Sync

timedatectl status

must show System clock synchronized: yes, NTP service: active. If not, install chrony, enable, verify chronyc tracking offset <0.1s, leap status Normal. Check virtualization time sync interference or firewall blocking UDP 123.

5.2 Port Conflicts

ss -lntp | grep -E ':(9090|9093|9100|8080|3000)\b'

should be empty. 9100 often used by CUPS on desktop distros.

5.3 Disk Mount & Filesystem

/data

must be separate block device (not root), ext4 or xfs (no NFS — mmap corruption risk), mount with noatime.

5.4 File Descriptor & Process Limits

ulimit -n

≥65536. systemd services need LimitNOFILE=65536 in unit file.

5.5 Kernel Parameters

Set in /etc/sysctl.d/99-prometheus.conf: net.ipv4.ip_local_port_range=10240 65000, net.ipv4.tcp_tw_reuse=1, net.core.somaxconn=4096, net.ipv4.tcp_max_tw_buckets=262144. Apply with sysctl --system. Don't enable tcp_tw_recycle (removed in 4.12+, breaks NAT).

5.6 Firewall & Security Groups

Prometheus egress to target scrape ports; targets ingress only from Prometheus IP; Alertmanager ingress only from Prometheus; Grafana ingress only from office/VPN. Example firewalld rich rule for node_exporter:

firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.20.30.11/32" port protocol="tcp" port="9100" accept'

.

5.7 SELinux

If Enforcing, label /data/prometheus with prometheus_var_lib_t via

semanage fcontext -a -t prometheus_var_lib_t "/data/prometheus(/.*)?"

and restorecon -Rv. If type missing, fallback to var_lib_t or generate policy via audit2allow.

5.8 DNS Resolution

If targets use hostnames, Prometheus resolves per scrape. Verify getent hosts and dig +short return correct IP <100ms. Use local DNS cache ( systemd-resolved or nscd).

5.9 Disk I/O Baseline

iostat -x 1 5

: new disk %util <5%, await <2ms. High baseline indicates bad disk or shared storage unsuitable for TSDB.

5.10 Offline Packages & Verification

Download binaries (Prometheus, Alertmanager, node_exporter) with curl -LO, verify sha256sum against official sha256sums.txt. Never run unverified binaries.

6. Deploy Prometheus Server

6.1 Deployment Choice

Binary+systemd: direct upgrades, clear paths, journalctl logs, no overhead. Good for ≤3 standalone instances. Container: unified image, GitOps-friendly, but data dir must be host volume, localhost!= host, reload via signal/API. Prometheus Operator: CRDs ( ServiceMonitor, PodMonitor, PrometheusRule) auto-generate config, reload; but abstraction layers complicate debugging, and managing non-K8s hosts requires extra config. Article uses binary for clarity.

6.2 Directory Layout

/usr/local/prometheus/          # binaries & config
├── prometheus
├── promtool
├── prometheus.yml
├── rules/
│   ├── host.rules.yml
│   ├── container.rules.yml
│   └── slo.rules.yml
└── console_libraries/

/data/prometheus/               # TSDB (dedicated disk)
/etc/prometheus/                # optional separate config dir

Separate config (human-edited) from data (Prometheus-owned) to avoid accidental chown corruption.

6.3 User & Permissions

useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
mkdir -p /usr/local/prometheus/rules /data/prometheus
chown -R prometheus:prometheus /data/prometheus
chown -R root:prometheus /usr/local/prometheus
chmod 750 /usr/local/prometheus /data/prometheus

Verify: data dir owned by prometheus, program dir root:prometheus (read-only).

6.4 Install Binaries

tar -xzf prometheus-2.53.0.linux-amd64.tar.gz
cd prometheus-2.53.0.linux-amd64
install -m 0755 -o root -g root prometheus /usr/local/prometheus/prometheus
install -m 0755 -o root -g root promtool /usr/local/prometheus/promtool
mkdir -p /usr/local/prometheus/consoles /usr/local/prometheus/console_libraries
cp -r consoles/* /usr/local/prometheus/consoles/
cp -r console_libraries/* /usr/local/prometheus/console_libraries/
/usr/local/prometheus/prometheus --version

Expect version output. Exec format error means wrong arch (use linux-arm64 for aarch64).

6.5 Main Config ( prometheus.yml )

Full config with global params, alerting, rule files, three scrape jobs (prometheus self, node via file_sd, cadvisor via kubernetes_sd). Key points: global.scrape_interval: 30s (business job overrides to 15s), scrape_timeout: 10s (< interval), evaluation_interval: 30s (aligned with scrape), external_labels for cluster/replica identification. alerting lists two Alertmanager endpoints (HA), api_version: v2 required for 0.27+. rule_files uses wildcard *.rules.yml (avoid loading .bak copies). node job uses file_sd_configs with refresh_interval: 60s and relabel to strip port from instance. cadvisor job uses kubelet proxy via API server (avoids direct 10250 cert issues), requires RBAC for nodes/metrics and nodes/proxy. insecure_skip_verify: false keeps cert validation.

6.6 systemd Unit

Unit includes --storage.tsdb.retention.time=15d and --storage.tsdb.retention.size=300GB (first triggered wins; size = 60% of disk, 40% headroom for compaction/WAL). --storage.tsdb.wal-compression reduces write amplification. --web.enable-lifecycle allows POST /-/reload (secure with network isolation). --web.enable-admin-api enables series deletion (disable if not needed). --query.timeout=2m, --query.max-concurrency=20. Hardening: ProtectSystem=full, ReadWritePaths=/data/prometheus.

6.7 Start & Verify

systemctl daemon-reload && systemctl enable --now prometheus

. Check systemctl status, journalctl -u prometheus on failure. Common errors: permission denied (dir ownership/SELinux), config file missing, YAML syntax, port conflict, storage corruption, fd limit. Pre-start config check: promtool check config. Post-start: ss -lntp | grep 9090, curl -s /-/healthy (expect "Prometheus Server is Healthy."), curl -s /-/ready (expect "Prometheus Server is Ready."), query /api/v1/targets for up=1.

6.8 Hot Reload Procedure

Never systemctl restart for config changes (breaks scrape, resets for timers). Instead: promtool check config, curl -X POST /-/reload, verify /api/v1/status/config returns success. On error, old config stays active. Restart only for CLI flag changes, version upgrades, TSDB corruption.

7. Host Metrics: node_exporter

7.1 Collectors

Default: cpu, meminfo, loadavg, filesystem, diskstats, netdev, netstat, stat, time, uname. Optional high-value collectors: systemd (service state, medium overhead), processes (process state counts, low), textfile (custom .prom files, negligible), interrupts (IRQ distribution, medium), ksmd / numa (memory merge/NUMA hit, low), mountstats (NFS latency, high if many NFS), buddyinfo / zoneinfo (memory fragmentation, low but many series). Enable only if needed for alerts/dashboards.

7.2 Install & systemd

useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter
install -m 0755 node_exporter /usr/local/bin/node_exporter
# unit file with --collector.systemd --collector.systemd.unit-include='(sshd|crond|chronyd|docker|containerd)\.service' --collector.processes --collector.textfile.directory=/var/lib/node_exporter/textfile --no-collector.ipvs
mkdir -p /var/lib/node_exporter/textfile && chown node_exporter:node_exporter /var/lib/node_exporter/textfile
systemctl daemon-reload && systemctl enable --now node_exporter

Note: --collector.systemd.unit-include regex must be single-quoted in systemd to avoid escape parsing. --web.listen-address=10.20.30.15:9100 to bind only internal IP.

7.3 Core Host Queries

CPU usage (exclude idle):

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

. Use avg for per-core average (0-100), not sum. Memory:

(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

(use MemAvailable, not MemFree). Disk:

(1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs",mountpoint!~\"/run.*|/var/lib/docker/.+\"} / node_filesystem_size_bytes{...}) * 100

(avail excludes root reserve). Disk IO utilization:

rate(node_disk_io_time_seconds_total{device=~"sd.*|nvme.*"}[5m]) * 100

(combine with read/write latency). Network:

rate(node_network_receive_bytes_total{device!~"lo|veth.*|docker.*|br-.*"}[5m]) * 8

(bits/sec, filter virtual interfaces). Load per core:

node_load1 / count by (instance) (node_cpu_seconds_total{mode="idle"})

(alert if >1.5 sustained).

7.4 textfile Collector for Custom Metrics

Scripts write .prom files atomically ( mktemp + mv), with trap cleanup, set -euo pipefail. Output 0 not omit metric (so absent() alerts work). Example: backup success timestamp, status, size. Alert if backup >26h old: (time() - backup_last_success_timestamp_seconds) > 93600 with humanizeTimestamp in annotation.

7.5 Common Host Collection Issues

Series spike: new mountpoints/network interfaces (often container overlay mounts). Drop via metric_relabel_configs on Prometheus side (e.g., drop node_filesystem_* with mountpoint=/var/lib/kubelet/.*). up flapping: check scrape_duration_seconds near timeout (exporter CPU starved) or network drops. Fix root cause (e.g., disable heavy systemd collector) not just increase timeout.

8. Container Metrics: cAdvisor to kube-state-metrics

8.1 Three Collection Paths

Path 1: kubelet built-in cAdvisor ( /metrics/cadvisor) — no extra deploy, fullest metrics (CPU throttling, memory working set), preferred for K8s. Access via API server proxy (reuses auth) or direct kubelet (needs cert). Path 2: standalone cAdvisor container (non-K8s Docker) — privileged, mount host paths read-only, bind 127.0.0.1:8080. Path 3: container runtime direct (containerd 1.7+ CRI) — coarser, version-dependent, avoid unless specific need.

8.2 Key Container Metric Fields & Pitfalls

Labels: container (name, POD for pause — exclude), name (cgroup path, volatile), id (cgroup ID, changes on restart), pod (pod name, Deployment pods change on rollout), image (full image:tag, latest or unique tags cause cardinality explosion), namespace (stable aggregation dimension). CPU usage:

sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{container!="",container!="POD",image!=""}[5m]))

(three filters mandatory). CPU % = usage / kube_pod_container_resource_limits{resource="cpu"} (from kube-state-metrics; no limit → no %). Memory: use container_memory_working_set_bytes (non-reclaimable, matches kubectl top), not usage (includes page cache). CPU throttling:

rate(container_cpu_cfs_throttled_periods_total)/rate(container_cpu_cfs_periods_total)

>25% indicates bursty limit, add replicas not just raise limit. Container restarts: increase(kube_pod_container_status_restarts_total[1h]) > 0. OOM:

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1

plus events kubectl get events --field-selector reason=OOMKilling.

8.3 kube-state-metrics: Object State, Not Resource Usage

Deploy via official manifests (ServiceAccount, ClusterRole, ClusterRoleBinding, Deployment, Service). Production tweaks: replicas: 2 with podAntiAffinity, resource limits (requests 100m/256Mi, limits 1/1Gi),

--metric-labels-allowlist=pods=[app,version],deployments=[app]

to drop high-churn labels like pod-template-hash. Verify series count 300-800; >5000 indicates label allowlist not working. Key queries: replica shortfall, pod not Running, PVC pending, node NotReady.

8.4 Pod Restart Series Breaks

Deployment pod names have random suffixes, causing series churn. Solutions: (1) Query-time aggregation via label_replace regex to extract workload prefix (works for Deployment, StatefulSet needs different regex). (2) Write-time workload label via recording rules (complex). (3) Switch to VictoriaMetrics/Thanos (architectural change). Note: rate() needs two samples, new pods have initial gap; use for: 2m in alerts.

9. Service Discovery: Auto-Scaling Target Lists

9.1 file_sd: Universal Intermediate

Prometheus reads JSON/YAML files generated externally (CMDB, Ansible, scripts). File format: array of objects with targets and labels. Generation script: fetch from CMDB API, validate minimum target count (prevent empty overwrite), atomic mv, token from file (600 perms). Run via cron/systemd timer every 5m; Prometheus refresh_interval: 60s picks up changes.

9.2 kubernetes_sd: Native K8s Discovery

Five roles: node (kubelet/node_exporter), pod (annotated pods), service / endpoints (service endpoints), ingress (blackbox probes). Recommended: pod role with annotations prometheus.io/scrape=true, prometheus.io/port, prometheus.io/path. Relabel config: keep only scraped pods, rewrite __address__ with port annotation, set __metrics_path__, labelmap pod labels (watch cardinality — exclude pod-template-hash).

9.3 Relabel Execution Order

relabel_configs

(pre-scrape, can modify __address__, __metrics_path__, keep/drop targets) → scrape → metric_relabel_configs (per-sample, modify/drop labels). Order matters: drop before keep kills later rules. Debug via Prometheus UI Targets page: compare Discovered vs Target Labels.

10. Business Metrics: Instrumentation to Pushgateway

10.1 Four Golden Signals & Naming

Latency, Traffic, Errors, Saturation. Naming:

<namespace>_<subsystem>_<name>_<unit>

(e.g., order_api_request_duration_seconds, order_api_requests_total, order_api_inflight_requests). Rules: counters end with _total, time in seconds (not milliseconds), avoid count / sum / avg in metric name, no instance/env in name (use labels).

10.2 Go Instrumentation

Use prometheus/client_golang. HistogramVec for latency with route template ( r.Pattern in Go 1.22+) to avoid high cardinality from dynamic paths. CounterVec for requests with method/route/status (status normalized to 2xx/4xx/5xx). Gauge for in-flight. Buckets aligned to SLA (dense around target percentile). Wrapper captures actual status code via custom ResponseWriter.

10.3 Java (Spring Boot + Micrometer)

Enable Actuator prometheus endpoint on separate management port (8081). management.metrics.tags.application adds app name. percentiles-histogram: true emits _bucket series for quantiles. slo config creates precise SLO buckets.

10.4 Python (prometheus_client)

Decorator-based tracking. Multi-process (gunicorn) requires prometheus_multiproc_dir + MultiProcessCollector or Pushgateway (not recommended for long-lived services).

10.5 Pushgateway: Only for Short-Lived Jobs

Deploy binary+systemd with --persistence.file and --persistence.interval=5m. Push via

curl --data-binary @- /metrics/job/<job>/instance/<instance>

. Limits: metrics never expire (must alert on last_success_timestamp), instance label required to avoid overwrites, delete via DELETE /metrics/job/.../instance/... (job-level delete is dangerous).

11. Label System & Cardinality Governance

11.1 Cardinality = Survival

Each unique label set = one series. Real incident: Nginx exporter added $request_uri with query string as label → series 50k→1.8M in 3 days, memory 6→26 GB, query timeouts. Detection: prometheus_tsdb_head_series trend, topk(10, sum by (job) ({__name__=~".+"})), count by (path) (nginx_http_requests_total) (run off-peak, with timeout).

11.2 Label Design Principles

Good labels: low-cardinality enums (status, method, region, env), aggregation dimensions (namespace, service, instance), correlation IDs (cluster, job). Bad labels: user/order/session IDs, full URLs/SQL/stack traces, timestamps/UUIDs, high-churn versions (put in image label with controlled release frequency). Gray area: instance (IP:port, scales with hosts), pod (churn on rollout).

11.3 metric_relabel_configs for Reduction

Drop high-cardinality labels at scrape: labeldrop (remove label, keep sample), drop (discard sample), replace (rewrite value, e.g., normalize /api/v1/orders/123 → /api/v1/orders/{id}). Risk: labeldrop can create duplicate samples (same timestamp, same labels) → both dropped.

11.4 Recording Rules for Pre-aggregation

Pre-compute frequent complex queries. Naming: level:metric:operation (e.g., instance:node_cpu_utilization:ratio, service:http_request_duration_p99:5m). Verify with promtool check rules and check lastEvaluation timestamp via API.

11.5 Cardinality Growth Monitoring

Track prometheus_tsdb_head_series and delta(prometheus_tsdb_head_series[24h]). Alert on >30% daily growth: delta(...) / (prometheus_tsdb_head_series offset 24h) > 0.3.

12. Alertmanager: From Rule to Notification

12.1 Deploy HA Cluster

Two nodes minimum. Unit: --cluster.listen-address=0.0.0.0:9094, --cluster.peer=other-node:9094, --data.retention=120h, --storage.path persistent. Verify cluster status via /api/v2/status (peers listed, status ready).

12.2 Config Deep Dive

route

: group_by: [alertname, cluster, namespace] (not instance — avoids 200 notifications for disk full). group_wait: 30s (allows inhibition), group_interval: 5m, repeat_interval: 4h. Sub-routes: severity=critical → oncall-pager (10s wait, 1h repeat), team=db → dba-team, severity=info → digest mail (5m wait, 24h repeat). continue: false default (stops at first match); set true to match multiple routes. inhibit_rules: NodeDown suppresses same-instance warning/critical; ServiceDown suppresses HighErrorRate / HighLatency for same service/namespace. equal must specify exact label list (empty = global suppression disaster). Receivers: webhook with send_resolved: true, email with templated subject.

12.3 Validation & Reload

amtool check-config

(checks inhibit rule count — indentation errors silently drop rules). Reload: curl -X POST /-/reload (atomic, but doesn't verify receiver connectivity).

12.4 amtool Troubleshooting

amtool alert

(list firing),

amtool config routes test --verify.receivers=... severity=critical alertname=NodeDown instance=...

(shows matched route — critical for "alert not received" cases), amtool silence query,

amtool silence add instance=... --duration=2h --comment=... --author=...

. Silences are unconditional — must have comment, author, expiry. Audit expired silences regularly.

13. Alert Rules: From Thresholds to Actionable Judgments

13.1 Good Alert Criteria

Actionable (recipient knows next step), owned (team label), evidence (annotations with value/threshold/links), deduplicated (inhibition), self-resolving (expr recovers). Bad: CPU > 80% (batch jobs hit 95% harmless). Better: CPU > 85% AND load_per_core > 2 for 10m.

13.2 Host Alerts

NodeDown

: up{job="node"}==0 for 2m (filters network blips). NodeRebooted:

node_boot_time_seconds - node_boot_time_seconds offset 15m > 0

(detects reboot even if up recovers). NodeMemoryPressure: instance:node_memory_utilization:ratio > 0.92 for 10m. NodeDiskWillFillIn4Hours: predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0 (linear regression beats static threshold). Pair with absolute NodeDiskAlmostFull (<2GB avail) for sudden fills.

13.3 Container Alerts

PodCrashLooping

: increase(kube_pod_container_status_restarts_total[15m]) > 3 for 5m. PodNotReady:

sum(kube_pod_status_ready{condition="true"})==0 AND kube_pod_status_phase{phase="Running"}==1

(excludes Pending). ContainerMemoryNearLimit: working_set / limit > 0.9 for 10m. ContainerCPUThrottling: throttled_periods / periods > 0.25 for 15m (add replicas, not just limit).

13.4 Business SLO Alerts

Error budget burn rate (SRE workbook multi-window). Recording rule: service:http_request_error_ratio:5m = 5xx rate / total rate. Alerts: fast burn (14.4× budget, 2m window) → critical; slow burn (6× budget, 30m window) → warning. Low-traffic guard: require rate(http_requests_total[5m]) > 1 and error ratio >5%.

13.5 for Duration Design

Table: NodeDown 2m, CPU/Mem 10m, Disk 15m, PodRestart 5m, BusinessError 2m, CertExpiry 1h. for works with evaluation_interval (30s → 2m needs 4 consecutive evaluations). Restart resets timer.

13.6 Alert Quality Review

Prometheus ALERTS gauge disappears on resolve — can't use increase(). Use ALERTS_FOR_STATE (since 2.20) for current firing duration. True trigger counts need Alertmanager webhook receiver logging to DB (alertname, instance, starts_at, ends_at). Weekly SQL: count fires, avg duration. If avg duration <300s and >80% of fires, alert too noisy — adjust or delete. Target: ≤5 alerts/person/day.

14. Grafana: Making Metrics Visible

14.1 Deploy & DataSource

PostgreSQL backend (not SQLite). Disable anonymous, integrate LDAP/OAuth. Provision datasource via file: access: proxy, url: http://prometheus:9090, timeInterval: 30s (matches scrape), httpMethod: POST, queryTimeout: 120s, manageAlerts: false, editable: false.

14.2 Dashboard Design Principles

Top row: global SLO attainment, error budget remaining, active alerts by severity, core QPS. Second row: host/container/business drill-down (Top N, anomaly counts). Third row: detailed instance tables with sorting/threshold coloring. Avoid: 30 tiny panels, pie charts for time series, non-zero Y-axis without note, 5m refresh on historical panels.

14.3 Common Panel Queries

Host top CPU: topk(20, instance:node_cpu_utilization:ratio * 100) (Table, percent unit, thresholds 70/85). Latency quantiles: three

histogram_quantile(0.50/0.95/0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

on same panel. Check +Inf bucket share — if P99 near it, add larger buckets. Error rate:

sum(rate(http_requests_total{status=~"5xx"}[5m])) / clamp_min(sum(rate(http_requests_total[5m])), 0.0001)

(avoid NaN).

14.4 Variables & Linking

Variables via label_values(up{job="node"}, cluster) with refresh: 2 (on time range change), multi: true. Use in queries: {cluster=~"$cluster"} (regex for multi-select). Performance: avoid label_values on high-cardinality metrics; use kube_node_info or recording rule.

15. High Availability & Long-Term Storage

15.1 Single-Node Limits

Active series ≤2M, targets ≤5k, no replica (blind spot on failure), local disk limits retention (15-30d). Three paths: dual replica + remote storage (medium scale, HA), Thanos (large scale, global view, long retention), VictoriaMetrics (high cardinality/write volume).

15.2 Dual Replica

Two Prometheus scrape same targets, each remote write to same long-term storage. Only config diff: external_labels.replica: prom-a / prom-b (mandatory for deduplication). Cost: double scrape load on targets, double local storage, Alertmanager deduplicates via fingerprint (ignores replica label).

15.3 remote_write Tuning

queue_config

: capacity: 10000, max_shards: 50, min_shards: 5, max_samples_per_send: 5000, batch_send_deadline: 10s, min_backoff: 100ms, max_backoff: 5s. Drop Prometheus self-metrics via write_relabel_configs. Risk: remote storage down → WAL grows unbounded → disk full. Monitor prometheus_remote_storage_samples_pending and failure rate; alert on >100k pending for 10m.

15.4 Thanos Overview

Sidecar (uploads blocks, serves recent queries), Store Gateway (reads historical blocks from object storage), Querier (aggregates, deduplicates via --query.replica-label=replica), Compactor (downsamples to 5m/1h blocks), Receive (optional remote write ingest). Missing --query.replica-label → duplicate series, doubled values.

15.5 Long-Term Storage Cost Control

Raw (15-30s) 15-30d, downsampled (5m) 1y, aggregated (1h) 3y. Thanos compactor auto-generates downsampled blocks. Prometheus native: recording rules with interval: 5m produce lower-frequency series (not true downsampling — raw still stored). To save space, drop raw metrics via write_relabel_configs at remote write (lose drill-down).

16. Security Hardening

16.1 Attack Surface

Prometheus 9090 (query, reload, admin API — unauthenticated can delete data/reload), Alertmanager 9093 (alerts, silences — can silence alerts), node_exporter 9100 (host info, kernel version, mounts), cAdvisor 8080 (container info, env vars), Grafana 3000 (all metrics), Pushgateway 9091 (metric write/delete).

16.2 Network Isolation

Exporters bind internal IP only. Prometheus→exporters via dedicated VLAN/security group. Grafana only office/VPN. Alertmanager 9093 only Prometheus+jump host. 9094 only Alertmanager peers. K8s NetworkPolicy example: allow Prometheus egress to prod namespace pods on 8080 and host IPs on 9100.

16.3 Application Auth

Prometheus 2.24+ supports web.yml with TLS and basic auth (bcrypt via htpasswd). Must add credentials to self-scrape job. node_exporter 1.5+ supports --web.config.file; cAdvisor lacks auth (rely on network).

16.4 Disable Dangerous Endpoints

Remove --web.enable-lifecycle if no dynamic reload needed. Remove --web.enable-admin-api if no series deletion needed. Unauthorized

DELETE /api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}

wipes all data; POST /-/reload reloads config.

16.5 Credential Management

Priority: 1) File mount + strict perms (600, service user). 2) Env vars (visible in /proc). 3) Vault/KMS via init container. Never hardcode in config files (leak via Git, images, backups).

16.6 Audit & Alerting

Alert on config reload frequency:

changes(prometheus_config_last_reload_success_timestamp_seconds[10m]) > 2

. Alert on silence creation: increase(alertmanager_silences{state="active"}[1h]) > 3 (requires Alertmanager self-scrape).

17. Performance Tuning

17.1 Scrape Side

Reduce targets, increase intervals (host 30s, business 15s, K8s objects 60s), drop series (disable collectors, metric_relabel_configs drop), drop labels ( labeldrop). scrape_timeout ≤ 1/3 scrape_interval to avoid cascade.

17.2 TSDB Parameters

Defaults --storage.tsdb.min-block-duration=2h, max-block-duration=36h fine. --storage.tsdb.head-chunks-write-queue-size (default 0 sync) can increase to 10000 for write throughput (risk: lose recent data on crash). Verify flag exists in version.

17.3 Query Side

Slow query patterns: large time range (30d) + high cardinality ( container_*) + complex agg; tiny rate window ( [1m] with 30s scrape = 2 points); Grafana Max data points too high (5000 → millions of points). Fix: use recording rules for long-range queries, set Grafana Max data points ~1000, Min step: 30s, disable instant queries.

17.4 Rule Computation Cost

200 rules × 30s = 400 queries/min. Optimize: merge shared base expressions, reduce short- for high-frequency rules, move ad-hoc aggregations to Grafana. Monitor prometheus_rule_group_last_duration_seconds >5s → optimize.

17.5 Real Memory Optimization Case

1.5M series, memory 12→28 GB. Top metric: http_request_duration_seconds_bucket 800k series (user_id label). Fix: metric_relabel_configs labeldrop user_id. Old series persist until retention; immediate cleanup via DELETE /api/v1/admin/tsdb/delete_series?match[]=... + POST /api/v1/admin/tsdb/clean_tombstones (dangerous: backup snapshot first, mv block to quarantine, delete WAL segments, restart, verify with promtool tsdb analyze). If corruption ( unexpected end of JSON input), identify bad block via promtool tsdb list/analyze, mv out, optionally dump salvage, restart. Memory release delayed by Go GC.

18. Troubleshooting Closed Loop: Six Real Cases

18.1 Batch Host Targets Down

40 hosts in one DC down. ICMP ok, TCP connection refused. ss -lntp shows stray nohup exporter holding port 9100, systemd service failed. Fix: identify/kill stray process, restart node_exporter, batch verify via Ansible. Root cause: untracked debug process in rc.local. Prevention: ban nohup, port conflict pre-check in systemd.

18.2 Prometheus OOM from Series Leak

Memory +1-2GB/day, OOM every 3 days. head_series 800k→2.1M. Top metric: container_network_receive_bytes_total 450k series from batch-jobs namespace (3000 pod labels vs 50 actual pods). Cause: short-lived Jobs create/destroy pods, series linger until retention. Fix: drop batch-jobs namespace metrics via metric_relabel_configs, longer-term drop unused container metrics, add job label to batch workloads. Verify growth slope flattens.

18.3 Disk Full → TSDB Corruption

Query 500 errors, no space left on device. Disk 100% (500GB). Cause: new high-cardinality metric (path label) tripled series; retention.size=350GB > usable space (460GB) + compaction/WAL headroom. Recovery: stop Prometheus, mv oldest blocks to quarantine (verify remote write has them), backup+clear WAL, restart, promtool tsdb analyze. If corruption, identify bad block via analyze, mv out, restart. Postmortem: retention.size = 60% disk, add disk prediction alert, cardinality review for new metrics, monthly promtool tsdb analyze.

18.4 Alert Firing But Not Received

Prometheus shows firing, Alertmanager silent. Check chain backward: receiver logs → Alertmanager received? → route match? → Prometheus push? Prometheus /api/v1/alertmanagers returns empty → prometheus_notifications_alertmanagers_discovered=0. Root cause: Alertmanager DNS record deleted, static config resolves only at startup. Fix: switch to IPs, reload. Prevention: use IPs, add independent health check (external curl+SMS) for alerting pipeline.

18.5 Missing Container Metrics for Namespace

New payment namespace absent. Targets healthy, kubelet metrics ok. Found metric_relabel_configs drop namespace=~'test-.*' but namespace named test-payment (non-standard). Fix: tighten regex to test-tmp-.*. Prevention: precise drop patterns, naming standards, evaluate new namespaces against existing rules.

18.6 Grafana Query Timeout

One panel 45s timeout (Gateway Timeout). Direct Prometheus query returns 2M points in 45s. Cause: 7d range, Grafana auto-step 15s (Min step empty, Max data points 5000), no recording rule. Fix: panel Min step=1m, Max data points=1000, use recording rule instance:container_cpu_usage:rate5m (5m interval). Load drops to 2s. Governance: template dashboards, enforce recording rules for high-freq panels, lower query.timeout to 60s.

19. Daily Operations & Maintenance

19.1 Daily Checklist (Scripted)

Process: systemctl is-active prometheus alertmanager → active; else journalctl

Ready: curl -s localhost:9090/-/ready → Ready; else TSDB

Targets: count(up==0) → 0; else lastError

Disk: df -h /data → <70%; else clean old blocks

Series: prometheus_tsdb_head_series → <1.5M; else high cardinality

Rule latency: prometheus_rule_group_last_duration_seconds → <5s; else optimize

Notify queue: prometheus_notifications_queue_length → <100; else Alertmanager

Remote write: prometheus_remote_storage_samples_pending → <10k; else remote storage

Scrape errors: rate(prometheus_target_scrapes_error_count_total[1h]) → <1%; else targets

Cert expiry: openssl x509 -enddate -noout -in cert.pem → >30d; else renew

Script runs all, exits with failure count, outputs OK/FAIL lines.

19.2 Backup Strategy

Config (daily, to Git): prometheus.yml, rules/, alertmanager.yml, Grafana provisioning. Data (weekly, optional): TSDB snapshot via POST /api/v1/admin/tsdb/snapshot (size = current data, needs extra space). If remote write exists, local snapshot lower priority. Grafana: pg_dump -Fc / pg_restore --clean --if-exists (destructive).

19.3 Version Upgrade

Pre-check: record version, validate config with new promtool, backup binary+config, confirm disk headroom. Steps: stop, install new binary, start, verify /-/ready, buildinfo version, target count. Rollback: stop, restore binary, start. Major version (2.x→3.x): TSDB forward compatible only; once 3.x writes new blocks, downgrade fails. Upgrade off-peak, observe 30 min.

19.4 Certificate/Credential Rotation

Generate new cert/key, sign with internal CA, verify, atomic mv (same paths), curl -X POST /-/reload (hot reload if web.config.file set).

20. Production Considerations

20.1 Change Management

Mandatory review for: new scrape jobs, metric_relabel changes, retention changes, alert rule changes, Prometheus upgrades, data deletion. Review: series delta estimate, affected panels/alerts, rollback plan, maintenance window.

20.2 Canary Rollout

Test Prometheus 24h → prod replica 1 (1h) → replica 2. Single node: off-peak with rollback ready. Alert rules: start with severity: info to test channel, promote after week.

20.3 Access Control

Roles: Oncall (Prom RO, AM silence, Grafana view, Git RO), Monitoring Lead (RW all, Git write), Dev (Prom RO, Grafana own panels), Audit (RO all). Implement: Prometheus/AM via oauth2-proxy, Grafana native RBAC, Git via PR+CI promtool check.

20.4 Alert Fatigue Management

Monthly Top 10 review: "what action was taken?" → delete unactionable. Merge same-service resource alerts into one. Set thresholds at historical P95. Replace thresholds with SLO burn rate. Automate maintenance silences via CI.

20.5 Documentation & Runbooks

Every alert annotation must have runbook link with: meaning, impact scope, troubleshooting steps (commands), common causes/fixes, escalation path. No runbook = burden on oncall.

21. Summary

Core pillars: Unified label system (cluster/namespace/service/instance across host/container/business), Controlled capacity & cardinality (pre-calc samples/storage/memory, monitor growth, zero tolerance for high-cardinality labels), Actionable alerts (each answers "what now?" — unactionable alerts waste attention, the scarcest resource). Rollout: weeks 1-2 core infra + host alerts; weeks 3-4 container metrics + workload dashboards; weeks 5-6 business instrumentation + SLO alerts; weeks 7-8 recording rules, capacity alerts, runbooks, fire drill. Final tip: maintain independent monitoring-of-monitoring (5m curl /-/ready + SMS) to detect total monitoring stack failure.