Hands‑On Guide to Building a Prometheus Monitoring System

Prometheus is an open‑source time‑series monitoring and alerting system originally created at SoundCloud; its creator Matt Proud brought ideas from Google’s Borg and Borgmon. It became the second CNCF project after Kubernetes in 2016.

Prometheus Overview

Multi‑dimensional data model – slice and dice metrics by instance, service, endpoint, method.

Operational simplicity – run a server anywhere without external storage.

Scalable collection & decentralized architecture – independent servers per team.

Powerful query language – PromQL for flexible alerts and dashboards.

Installation

Two common ways:

Binary (out‑of‑the‑box)

$ wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz
$ tar xvfz prometheus-2.4.3.linux-amd64.tar.gz
$ cd prometheus-2.4.3.linux-amd64
$ ./prometheus --version
prometheus, version 2.4.3 (branch: HEAD, revision: 167a4b4e73a8eca8df648d2d2043e21bdb9a7449)

Run the server:

$ ./prometheus --config.file=prometheus.yml

Docker

$ sudo docker run -d -p 9090:9090 prom/prometheus

Optionally mount a local config file:

$ sudo docker run -d -p 9090:9090 \
    -v ~/docker/prometheus/:/etc/prometheus/ \
    prom/prometheus

Configuration File (prometheus.yml)

# my global config
global:
  scrape_interval: 15s   # How often to scrape targets.
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

rule_files:
  - "alert.rules"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

PromQL Basics

Instant vector selector example: up{job="prometheus"} Range vector selector (last 5 minutes): http_requests_total[5m] Common functions:

# per‑second average rate (slow counters)
rate(http_requests_total[5m])
# per‑second instant rate (fast counters)
irate(http_requests_total[5m])

HTTP API

GET /api/v1/query

GET /api/v1/query_range

GET /api/v1/series

GET /api/v1/label/<label_name>/values

GET /api/v1/targets

GET /api/v1/rules

GET /api/v1/alerts

GET /api/v1/targets/metadata

GET /api/v1/alertmanagers

GET /api/v1/status/config

GET /api/v1/status/flags

Grafana Integration

Install Grafana (Docker example): $ docker run -d -p 3000:3000 grafana/grafana Add a Prometheus data source with URL http://localhost:9090, then import dashboards such as “Prometheus 2.0 Stats” (ID 405) to visualise metrics.

Exporters

Node Exporter (Linux)

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz
$ tar xvfz node_exporter-0.16.0.linux-amd64.tar.gz
$ cd node_exporter-0.16.0.linux-amd64
$ ./node_exporter

Verify with curl http://localhost:9100/metrics and add the target to scrape_configs.

MySQL Exporter

$ wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.11.0/mysqld_exporter-0.11.0.linux-amd64.tar.gz
$ tar xvfz mysqld_exporter-0.11.0.linux-amd64.tar.gz
$ cd mysqld_exporter-0.11.0.linux-amd64
$ export DATA_SOURCE_NAME='root:123456@(192.168.0.107:3306)/'
$ ./mysqld_exporter

Nginx Exporter

Use the Nginx VTS exporter (exposes /status/format/prometheus) or other community exporters to collect HTTP server metrics.

JMX Exporter

$ wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar
$ java -javaagent:jmx_prometheus_javaagent-0.3.1.jar=9404:config.yml -jar myapp.jar

Metrics become available at http://localhost:9404/metrics.

Alerting

Create alert.rules:

groups:
- name: example
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."
  - alert: APIHighRequestLatency
    expr: api_http_request_latencies_second{quantile="0.5"} > 1
    for: 10m
    annotations:
      summary: "High request latency on {{ $labels.instance }}"
      description: "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)"

Reload Prometheus, then view rules at http://localhost:9090/rules and active alerts at http://localhost:9090/alerts.

Alertmanager

$ wget https://github.com/prometheus/alertmanager/releases/download/v0.15.2/alertmanager-0.15.2.linux-amd64.tar.gz
$ tar xvfz alertmanager-0.15.2.linux-amd64.tar.gz
$ cd alertmanager-0.15.2.linux-amd64
$ ./alertmanager

Configure Prometheus to send alerts to Alertmanager (add to prometheus.yml):

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["192.168.0.107:9093"]

Example alertmanager.yml with a webhook receiver:

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'

receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:5001/'

Advanced Topics

Service discovery (file, Kubernetes, Consul, etc.) lets Prometheus automatically find targets, avoiding manual scrape_configs updates.

Pushgateway is useful for short‑lived batch jobs that cannot be scraped reliably; push metrics to http://pushgateway:9091 before they disappear.

Conclusion

By combining Prometheus, Grafana, and Alertmanager you can build a full‑featured, cloud‑native monitoring stack that scales with micro‑service architectures, supports rich queries via PromQL, and provides flexible alert routing.