Master Prometheus: From Installation to Advanced Monitoring with Grafana

1. Overview of Prometheus

Prometheus is an open‑source monitoring and alerting system built on a time‑series database. It was created at SoundCloud in 2012 by Matt Proud, who drew inspiration from Google’s Borg and Borgmon. Designed for micro‑service architectures, Prometheus integrates tightly with containers and cloud‑native environments.

2. Core Requirements

Multi‑dimensional data model for flexible slicing and dicing. Operational simplicity – a single binary can run on a workstation without external storage. Scalable, decentralized data collection allowing independent monitoring servers. Powerful query language (PromQL) for alerting and graphing.

These four characteristics make Prometheus both a monitoring system and a time‑series database.

3. Installing Prometheus Server

3.1 Binary (stand‑alone) installation

$ wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz
$ tar xvfz prometheus-2.4.3.linux-amd64.tar.gz
$ cd prometheus-2.4.3.linux-amd64
$ ./prometheus --version

Run the server:

$ ./prometheus --config.file=prometheus.yml

3.2 Docker installation

$ sudo docker run -d -p 9090:9090 prom/prometheus

Mount a custom configuration if needed:

$ sudo docker run -d -p 9090:9090 \
  -v ~/docker/prometheus/:/etc/prometheus/ \
  prom/prometheus

4. Configuring Prometheus

The default prometheus.yml contains four sections:

global : scrape interval, evaluation interval, etc.

alerting : Alertmanager configuration.

rule_files : alerting rule files.

scrape_configs : target definitions (e.g., the Prometheus server itself).

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

rule_files:
  - "alert.rules"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

5. Learning PromQL

PromQL queries consist of metric names, optional label selectors, and functions. Examples:

# Simple up check
up

# Filter by job
up{job="prometheus"}

# Range vector (last 5 minutes)
http_requests_total[5m]

# Rate calculation
rate(http_requests_total[5m])

Aggregations such as sum, avg, max, and functions like irate, ceil are also supported.

6. Exporters – Collecting Metrics

Prometheus pulls metrics from HTTP endpoints exposed by exporters. Common exporters include:

node_exporter : system metrics (CPU, memory, disk).

mysqld_exporter : MySQL performance metrics.

nginx_exporter / VTS exporter : Nginx statistics.

jmx_exporter : Java application metrics via JMX.

Example: installing and running node_exporter on a Linux host.

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz
$ tar xvfz node_exporter-0.16.0.linux-amd64.tar.gz
$ cd node_exporter-0.16.0.linux-amd64
$ ./node_exporter

Verify with curl http://localhost:9100/metrics and add the target to scrape_configs:

scrape_configs:
  - job_name: 'server'
    static_configs:
      - targets: ['192.168.0.107:9100']

7. Visualizing with Grafana

Grafana provides rich dashboards for Prometheus data. $ docker run -d -p 3000:3000 grafana/grafana Configure a Prometheus data source (URL http://localhost:9090) and import community dashboards (e.g., ID 405 for node exporter).

8. Alerting

Define alert rules in a separate file (e.g., alert.rules) and reference it in prometheus.yml. Example rules:

groups:
- name: example
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Instance {{ $labels.instance }} down"
  - alert: APIHighRequestLatency
    expr: api_http_request_latencies_second{quantile="0.5"} > 1
    for: 10m
    annotations:
      summary: "High request latency on {{ $labels.instance }}"

Install Alertmanager and configure Prometheus to send alerts to it:

$ wget https://github.com/prometheus/alertmanager/releases/download/v0.15.2/alertmanager-0.15.2.linux-amd64.tar.gz
$ tar xvfz alertmanager-0.15.2.linux-amd64.tar.gz
$ ./alertmanager

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['192.168.0.107:9093']

Alertmanager’s alertmanager.yml defines receivers (email, webhook, etc.) and routing rules.

9. Advanced Topics

9.1 Service Discovery

Instead of static target lists, Prometheus can discover targets via Kubernetes, Consul, DNS, file‑based SD, and other mechanisms.

9.2 Pushgateway

Use Pushgateway for short‑lived batch jobs that finish before Prometheus can scrape them.

9.3 Dynamic Alert Management

While Prometheus does not expose an API for modifying rules at runtime, configuration management tools (Ansible, Chef, Puppet, Salt) are recommended for automated updates.

10. Summary

Prometheus, together with Grafana and Alertmanager, forms a complete cloud‑native monitoring stack. Its pull‑based architecture, powerful query language, and extensive exporter ecosystem make it suitable for Kubernetes, Docker, and traditional server environments. Proper configuration, service discovery, and alert routing are essential for production use.