Master Prometheus: Install, Configure, Query, and Alert with Grafana

Prometheus Overview

Prometheus is an open‑source monitoring and alerting system built on a time‑series database, originally created at SoundCloud in 2012 by Matt Proud, who drew inspiration from Google’s Borg and Borgmon. It fits naturally into micro‑service and container environments and became the second project to join the Cloud Native Computing Foundation in 2016.

Key Features

Multidimensional data model

Easy deployment and maintenance

Flexible data collection via Exporters

Powerful query language (PromQL)

These features make Prometheus both a monitoring system and a time‑series database.

Architecture Diagram

The core component is the Prometheus server, which scrapes metrics, stores them, evaluates expressions, and generates alerts. Optional components include Pushgateway, Alertmanager, and the Web UI.

Installing Prometheus Server

Binary (stand‑alone) method

$ wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz
$ tar xvfz prometheus-2.4.3.linux-amd64.tar.gz
$ cd prometheus-2.4.3.linux-amd64
$ ./prometheus --version
$ ./prometheus --config.file=prometheus.yml

Docker method

$ sudo docker run -d -p 9090:9090 prom/prometheus
# with custom config
$ sudo docker run -d -p 9090:9090 \
    -v ~/docker/prometheus/:/etc/prometheus/ \
    prom/prometheus

Prometheus Configuration

The default prometheus.yml consists of four main blocks:

global : global settings such as scrape_interval and evaluation_interval.

alerting : configuration for Alertmanager.

rule_files : list of alert rule files.

scrape_configs : targets to scrape, e.g. the Prometheus server itself.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
rule_files:
  - "alert.rules"
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

Learning PromQL

PromQL is the query language used on the Graph page and via the HTTP API. Data points consist of a metric name and optional labels, forming a time series. Metric types include Counter, Gauge, Histogram, and Summary.

# Example: list all up targets
up
# Filter by label
up{job="prometheus"}
# Range vector (last 5 minutes)
http_requests_total[5m]
# Rate functions
rate(http_requests_total[5m])
irate(http_requests_total[5m])

HTTP API Endpoints

GET /api/v1/query
GET /api/v1/query_range
GET /api/v1/series
GET /api/v1/label/<label_name>/values
GET /api/v1/targets
GET /api/v1/rules
GET /api/v1/alerts
GET /api/v1/status/config
GET /api/v1/status/flags
POST /api/v1/admin/tsdb/snapshot
POST /api/v1/admin/tsdb/delete_series
POST /api/v1/admin/tsdb/clean_tombstones

Installing Grafana

$ docker run -d -p 3000:3000 grafana/grafana

After starting Grafana, add a Prometheus data source with Name: prometheus, Type: Prometheus, URL: http://localhost:9090, Access: Browser. Grafana provides ready‑made dashboards for Prometheus metrics.

Collecting Metrics with Exporters

Node Exporter

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz
$ tar xvfz node_exporter-0.16.0.linux-amd64.tar.gz
$ cd node_exporter-0.16.0.linux-amd64
$ ./node_exporter
$ curl http://localhost:9100/metrics

Add the node exporter to scrape_configs and reload Prometheus.

MySQL Exporter

$ wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.11.0/mysqld_exporter-0.11.0.linux-amd64.tar.gz
$ tar xvfz mysqld_exporter-0.11.0.linux-amd64.tar.gz
$ cd mysqld_exporter-0.11.0.linux-amd64
$ export DATA_SOURCE_NAME='root:123456@(127.0.0.1:3306)/'
$ ./mysqld_exporter

Nginx Exporter

Two options: Lua‑based prometheus.lua or the Nginx VTS exporter that scrapes /status/format/json (or /status/format/prometheus in newer versions).

JMX Exporter

$ wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar
# Run Java app with agent
$ java -javaagent:jmx_prometheus_javaagent-0.3.1.jar=9404:config.yml -jar myapp.jar
$ curl http://localhost:9404/metrics

Alerting and Notification

Alert Rules

rule_files:
  - "alert.rules"

# alert.rules
groups:
- name: example
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."
  - alert: APIHighRequestLatency
    expr: api_http_request_latencies_second{quantile="0.5"} > 1
    for: 10m
    annotations:
      summary: "High request latency on {{ $labels.instance }}"
      description: "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)"

Alertmanager Setup

$ wget https://github.com/prometheus/alertmanager/releases/download/v0.15.2/alertmanager-0.15.2.linux-amd64.tar.gz
$ tar xvfz alertmanager-0.15.2.linux-amd64.tar.gz
$ cd alertmanager-0.15.2.linux-amd64
$ ./alertmanager

Configure Prometheus to point to Alertmanager:

alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets: ['192.168.0.107:9093']

Example alertmanager.yml with a webhook receiver:

global:
  resolve_timeout: 5m
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:5001/'

Advanced Topics

Service discovery can automatically populate scrape targets (e.g., Kubernetes, Consul, DNS). Pushgateway is useful for short‑lived batch jobs that cannot be scraped reliably.

Conclusion

Prometheus has rapidly become the de‑facto monitoring solution for cloud‑native environments, offering a flexible data model, powerful query language, and seamless integration with Grafana and Alertmanager, making it ideal for modern micro‑service architectures.