Master Prometheus: From Metrics Collection to Alerting and Visualization

Introduction

Prometheus is an open‑source, full‑stack monitoring solution. It provides metric exposition, scraping, storage, querying, visualization, and alerting, allowing you to gain insight into system health and quickly locate problems.

Ecosystem Overview

Prometheus consists of components for exposing metrics, scraping them, storing them in a built‑in time‑series database, querying with PromQL, visualizing via its own Web UI or Grafana, and sending alerts through Alertmanager.

Metric Exposure

Each monitored service is a Job . Services can expose metrics directly via an SDK or through exporters (e.g., MySQL, Consul). For short‑lived jobs, the Pushgateway can be used to push metrics.

Metric Scraping

Prometheus primarily uses the Pull model: it periodically scrapes the

/metrics

endpoint of targets. The default scrape interval is one minute and can be configured via

scrape_interval

in

prometheus.yml

.

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

Storage and Query

Scraped metrics are stored as time‑series in an internal TSDB. PromQL is used to query these series, either instantly or over a range.

Alerting

Alertmanager receives alerts generated from PromQL expressions. Alerts can be routed to email, Slack, etc., and support silencing and grouping.

Configuration Details

Service Registration

Jobs can be registered statically in

scrape_configs

or dynamically via service discovery (Consul, DNS, Kubernetes, etc.). Example static registration:

scrape_configs:
  - job_name: "node_export_consul"
    metrics_path: /node_metrics
    scheme: http
    consul_sd_configs:
      - server: localhost:8500
        services:
          - node_exporter

Dynamic Registration Note

When using dynamic registration, ensure

metrics_path

is set correctly; otherwise Prometheus may report an "INVALID" start token error.

Configuration Reload

Start Prometheus with

--web.enable-lifecycle

to allow runtime reloads via

POST /-/reload

.

prometheus --config.file=/usr/local/etc/prometheus.yml --web.enable-lifecycle
curl -v --request POST 'http://localhost:9090/-/reload'

Metric Types

Prometheus defines four metric types:

Counter : monotonically increasing (e.g., request counts).

Gauge : can go up and down (e.g., memory usage).

Histogram : bucketed distribution for observations.

Summary : pre‑computed quantiles.

Exporting Metrics

Use the official

client_golang

library to expose metrics. A minimal exporter:

package main
import (
  "net/http"
  "github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
  http.Handle("/metrics", promhttp.Handler())
  http.ListenAndServe(":8080", nil)
}

Custom counters, gauges, histograms, and summaries can be defined and registered:

myCounter := prometheus.NewCounter(prometheus.CounterOpts{Name: "my_counter_total", Help: "custom counter"})
myGauge := prometheus.NewGauge(prometheus.GaugeOpts{Name: "my_gauge_num", Help: "custom gauge"})
myHistogram := prometheus.NewHistogram(prometheus.HistogramOpts{Name: "my_histogram_bucket", Help: "custom histogram", Buckets: []float64{0.1,0.2,0.3,0.4,0.5}})
mySummary := prometheus.NewSummary(prometheus.SummaryOpts{Name: "my_summary_bucket", Help: "custom summary", Objectives: map[float64]float64{0.5:0.05,0.9:0.01,0.99:0.001}})
prometheus.MustRegister(myCounter, myGauge, myHistogram, mySummary)

Metrics with labels use

NewCounterVec

(or similar) and

With(prometheus.Labels{...})

to set values.

myCounterVec := prometheus.NewCounterVec(prometheus.CounterOpts{Name: "my_counter_total", Help: "custom counter"}, []string{"label1", "label2"})
myCounterVec.With(prometheus.Labels{"label1":"1", "label2":"2"}).Inc()

PromQL Basics

PromQL expressions can be instant vectors, range vectors, scalars, or strings. Common functions include

rate()

,

irate()

,

sum()

with

by()

or

without()

, and

histogram_quantile()

for percentile calculations.

# Example: QPS per path
sum(rate(demo_api_request_duration_seconds_count{job="demo",method="GET",status="200"}[5m])) by (path)

Grafana Visualization

Grafana can be added as a data source pointing to Prometheus, then dashboards are built using PromQL queries.

Alertmanager

Define alert rules in a separate file and reference it from

prometheus.yml

. Example rule triggers when a job has no up targets for one minute:

groups:
- name: simulator-alert-rule
  rules:
  - alert: HttpSimulatorDown
    expr: sum(up{job="http_srv"}) == 0
    for: 1m
    labels:
      severity: critical

Configure Alertmanager with SMTP settings to send email notifications, and use its UI to silence alerts.

Summary

This guide provides a complete overview of Prometheus, covering its architecture, metric collection models, configuration, custom exporters, query language, visualization with Grafana, and alerting with Alertmanager, equipping developers and operations engineers to monitor and observe their systems effectively.