Prometheus Deep Dive: The De Facto Standard for Cloud‑Native Monitoring

Overview

This guide presents a complete analysis of Prometheus, from its design philosophy to practical deployment in a multi‑cloud environment. It starts with a concrete migration story, compares Prometheus with traditional monitoring tools, and then details every component of the monitoring stack.

Migration Story

Developer "A" inherited a Zabbix setup that required manual edits to static_configs in prometheus.yml for each new service, causing frequent reloads. After moving to Kubernetes, service discovery broke, and the single‑node TSDB could not retain data long enough, prompting the addition of Thanos for long‑term storage.

After refactoring, the system achieved 95% alert accuracy and reduced MTTR from 30 minutes to 8 minutes.

Why Prometheus?

Prometheus adopts a pull model, giving the server control over scrape intervals, built‑in health checking via the up metric, and a decentralized architecture without a single point of failure. The trade‑offs include the inability to monitor short‑lived jobs directly, which is solved by the optional Pushgateway.

Core Architecture

Prometheus Server : discovers targets, scrapes /metrics, stores data in a local TSDB, evaluates rules, and serves PromQL queries.

Pushgateway : transient storage for short‑lived batch jobs.

Alertmanager : de‑duplicates, groups, routes, and silences alerts.

Exporters : expose metrics from infrastructure components (node_exporter, mysqld_exporter, etc.).

Metric Types

Four metric families are defined:

Counter – monotonically increasing, e.g., http_requests_total.

Gauge – arbitrary values, e.g., CPU usage.

Histogram – bucketed counts with a sum and count, suitable for latency distribution and cross‑instance aggregation.

Summary – client‑side quantiles; not aggregatable across instances.

The article recommends using Histograms for most cases because they support cross‑instance aggregation, whereas Summaries are limited to single‑instance use.

PromQL Essentials

Key concepts include instant vectors, range vectors, and scalar values. Common functions are rate(), irate(), increase(), histogram_quantile(), and aggregation operators such as sum by (job). Example queries:

# 5‑minute QPS per job
sum by (job) (rate(http_requests_total[5m]))

# P99 latency for a job
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))

# Error rate > 5%
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
  / sum by (job) (rate(http_requests_total[5m])) > 0.05

Service Discovery

Prometheus supports several discovery mechanisms:

Static – hard‑coded target list.

Consul SD – pulls services from Consul.

Kubernetes SD – discovers Pods, Services, Endpoints; filters via annotations.

EC2/Alibaba Cloud SD – uses cloud provider APIs.

file_sd – reads JSON/YAML files generated by external scripts.

Relabeling ( relabel_configs) modifies target labels before scraping, while metric_relabel_configs rewrites metric labels after collection. Actions include replace, keep, drop, labelmap, and labeldrop.

Remote Storage & High Availability

Local TSDB limits retention (default 15 days), scalability, and HA. Remote storage solves these issues. The article compares three solutions:

Thanos – sidecar + object storage; provides global query, compaction, and HA via multiple sidecars.

Mimir – micro‑service architecture with strong multi‑tenant support.

VictoriaMetrics – single binary (or cluster) with optional object storage and MetricsQL extensions.

Selection criteria include existing Kubernetes + object storage (Thanos), massive SaaS multi‑tenant workloads (Mimir), or moderate scale with performance focus (VictoriaMetrics).

Recording & Alerting Rules

Complex PromQL expressions can be pre‑computed with Recording Rules, reducing query latency from seconds to milliseconds. Naming convention <level>:<metric>:<operation> (e.g., job:http_requests:rate5m) aids readability.

Alerting Rules define when an expression should fire, how long it must stay true ( for), severity labels, and human‑readable annotations. Alertmanager routes alerts based on severity, groups them, and can suppress lower‑severity alerts when a critical one is active.

Best‑Practice Checklist

Use clear metric names: [namespace]_[subsystem]_[name]_[unit] and add _total for Counters.

Design low‑cardinality labels (job, instance, method, status); avoid user_id, trace_id.

Histogram buckets should cover SLO thresholds and P99, with 10‑15 buckets.

Set scrape_interval between 15 s and 30 s for most services; longer for low‑frequency checks.

Estimate storage: time_series × (86400 / scrape_interval) × 1.5 B per day.

Prefer Kubernetes SD in cloud‑native clusters; use file_sd or EC2_sd for external environments.

Adopt Thanos (or Mimir/VictoriaMetrics) for >15‑day retention and HA.

Leverage Recording Rules for heavy queries; keep query windows narrow.

Configure Alertmanager with group_by, group_wait, group_interval, repeat_interval, and inhibit_rules to avoid alert storms.

Spring Boot Integration

Spring Boot exposes metrics via micrometer-registry-prometheus at /actuator/prometheus. Example application.yml config enables health, info, and Prometheus endpoints, sets common tags, and defines histogram buckets and SLOs. Custom counters, timers, and gauges are registered as beans and used in business code.

Prometheus scrape config for the Spring Boot service:

scrape_configs:
  - job_name: 'gateway-service'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 15s
    static_configs:
      - targets: ['gateway-1:8080', 'gateway-2:8080']
        labels:
          env: 'prod'
          region: 'huadong'

Common Pitfalls & Remedies

Negative rate values : caused by Counter reset after restart; newer Prometheus versions handle this automatically.

Inaccurate histogram_quantile : ensure P99 falls inside a non‑ +Inf bucket; redesign buckets or use Micrometer’s auto‑bucket feature.

Scrape timeouts : increase scrape_interval or reduce /metrics work; keep scrape_timeout < scrape_interval.

High‑cardinality labels : drop them via metric_relabel_configs or remove from application code.

Alert storms : use group_by, inhibit_rules, and appropriate for durations.

Slow PromQL queries : create Recording Rules, narrow time ranges, limit aggregation dimensions, and upgrade Prometheus.

Disk exhaustion for long retention : offload historic data to Thanos/Mimir/VictoriaMetrics and enable compaction.

Interview Self‑Test

The article concludes with a set of interview questions covering Pull vs Push, metric types, PromQL functions, service discovery, remote storage choices, Recording Rules, and handling high‑cardinality metrics.