Comprehensive Guide to Prometheus: Installation, Configuration, PromQL, Exporters, Grafana, and Alerting

Prometheus is an open‑source, time‑series based monitoring and alerting system originally created at SoundCloud to overcome the limitations of traditional monitoring tools in large micro‑service environments. It offers a multi‑dimensional data model, operational simplicity, scalable decentralized data collection, and a powerful query language (PromQL).

The default Prometheus configuration file ( prometheus.yml) consists of four main blocks: global (scrape and evaluation intervals), alerting (Alertmanager settings), rule_files (alerting rules), and scrape_configs (targets to scrape). Example configuration snippets are shown below:

<span># Global settings</span>
global:
  scrape_interval: 15s
  evaluation_interval: 15s

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

# Rule files
rule_files:
  - "alert.rules"

# Scrape targets
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Installation can be done by downloading the pre‑compiled binary and running it directly, or by using Docker:

<span># Binary installation</span>
$ wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz
$ tar xvfz prometheus-2.4.3.linux-amd64.tar.gz
$ ./prometheus --config.file=prometheus.yml

<span># Docker installation</span>
$ docker run -d -p 9090:9090 prom/prometheus

PromQL allows instant‑vector selectors (e.g., up{job="prometheus"}) and range‑vector selectors (e.g., http_requests_total[5m]). Common functions include rate(), irate(), count(), sum(), and aggregation operators such as topk. The web UI provides a Graph page for visual queries and a Console view for raw results.

Grafana is recommended for richer dashboards. After installing Grafana (e.g., docker run -d -p 3000:3000 grafana/grafana), add a Prometheus data source with URL http://localhost:9090. Pre‑built dashboards (e.g., ID 405 for Node Exporter) can be imported to visualize server, MySQL, Nginx, and other metrics.

Metrics are collected via exporters. The node_exporter gathers host metrics, mysqld_exporter exposes MySQL statistics, and the JMX exporter collects Java application metrics. Example exporter launch commands:

<span># Node exporter</span>
$ wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz
$ tar xvfz node_exporter-0.16.0.linux-amd64.tar.gz
$ ./node_exporter

<span># MySQL exporter</span>
$ wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.11.0/mysqld_exporter-0.11.0.linux-amd64.tar.gz
$ tar xvfz mysqld_exporter-0.11.0.linux-amd64.tar.gz
$ export DATA_SOURCE_NAME='root:password@(localhost:3306)/'
$ ./mysqld_exporter

Alerting is handled by defining rules in alert.rules and sending alerts to Alertmanager. A simple rule example:

groups:
- name: example
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

Alertmanager is started similarly to Prometheus and configured (in alertmanager.yml) with receivers such as email, Slack, or custom webhooks. Example webhook receiver configuration:

global:
  resolve_timeout: 5m

route:
  receiver: 'web.hook'

receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:5001/'

Service discovery (SD) mechanisms (Kubernetes, Consul, DNS, file‑based, etc.) can automatically populate scrape_configs without manual target lists. For short‑lived jobs, the Pushgateway can be used to push metrics before they disappear.

The guide concludes with references to further reading on Prometheus, Cloud Native monitoring, and related time‑series databases.