Master Prometheus: From Metrics Collection to Alerting and Visualization

Introduction

Prometheus is an open‑source, full‑stack monitoring solution that provides metric collection, storage, visualization, and alerting.

Overall Architecture

Prometheus consists of components for metric exposition, scraping, storage, querying, and alerting. Services expose metrics via an HTTP endpoint or exporters (e.g., MySQL, Consul). Prometheus scrapes these metrics using a Pull model (default 1‑minute interval) or via PushGateway for short‑lived jobs.

Metric Exposure

Each monitored service is a Job with one or more targets . Metrics can be exported using the official SDK or community exporters. PushGateway allows services to push metrics actively.

Scraping and Storage

Scraping is configured in scrape_configs within prometheus.yml. Example static configuration:

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

Dynamic registration can use service discovery mechanisms such as Consul, Kubernetes, DNS, etc. Example Consul configuration:

- job_name: "node_export_consul"
  metrics_path: /node_metrics
  scheme: http
  consul_sd_configs:
    - server: localhost:8500
      services:
        - node_exporter

Scraped metrics are stored in an internal time‑series database and flushed to disk every two hours, with a write‑ahead log for crash recovery.

Metric Model

Each time series consists of a metric name with label set, a timestamp, and a sample value. Example format:

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 1027

Prometheus defines four metric types: counter , gauge , histogram , and summary .

Custom Exporter in Go

Use the client_golang library to expose metrics. Minimal exporter:

package main
import (
  "net/http"
  "github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
  http.Handle("/metrics", promhttp.Handler())
  http.ListenAndServe(":8080", nil)
}

Define custom counters, gauges, histograms, and summaries, register them, and update values in handlers. Example with labels:

myCounter := prometheus.NewCounterVec(prometheus.CounterOpts{
  Name: "my_counter_total",
  Help: "custom counter",
}, []string{"label1", "label2"})
myCounter.With(prometheus.Labels{"label1":"1", "label2":"2"}).Inc()

PromQL Queries

PromQL provides instant vectors, range vectors, and aggregation functions. Examples:

Instant query: go_gc_duration_seconds_count Label filter: go_gc_duration_seconds_count{instance="127.0.0.1:9600"} Regex filter: go_gc_duration_seconds_count{instance=~"localhost.*"} Range query: go_gc_duration_seconds_count[5m] Rate: rate(http_requests_total[5m]) Instant rate: irate(http_requests_total[5m]) Aggregation: sum(rate(http_requests_total[5m])) by (path) Histogram quantiles can be calculated with histogram_quantile(0.5, my_histogram_bucket), but bucket boundaries must be chosen carefully to reduce estimation error.

Grafana Visualization

Connect Grafana to Prometheus as a data source, create dashboards, and use PromQL expressions in panels to visualize metrics.

Alerting with Alertmanager

Alertmanager receives alerts from Prometheus, groups them, and forwards them via email, Slack, etc. Example alert rule:

groups:
- name: simulator-alert-rule
  rules:
  - alert: HttpSimulatorDown
    expr: sum(up{job="http_srv"}) == 0
    for: 1m
    labels:
      severity: critical

Configure Alertmanager in prometheus.yml and define receivers (e.g., SMTP). Alerts transition from PENDING to FIRING after the defined duration, then notifications are sent.

Images illustrating the concepts are retained below: