Master Prometheus: From Setup to Advanced Monitoring in Cloud‑Native Environments

Introduction

Prometheus is an open‑source time‑series monitoring and alerting system originally created at SoundCloud in 2012 by Matt Proud, who previously worked on Borg and Borgmon at Google. It was designed to overcome the limitations of traditional monitoring stacks such as StatsD and Graphite in large micro‑service environments.

Relation to Cloud‑Native

Prometheus became the second project accepted by the Cloud Native Computing Foundation (CNCF) in 2016, shortly after Kubernetes, and is tightly coupled with container‑based deployments and service‑discovery mechanisms.

Key Features

Multi‑dimensional data model allowing flexible slicing by instance, service, endpoint, etc.

Operational simplicity – a single binary can be started on any host without external storage.

Decentralised architecture that scales horizontally and lets teams run independent servers.

Powerful PromQL query language for ad‑hoc exploration and alerting.

Installation

Binary (stand‑alone)

Download the latest release, extract it and start the server:

$ wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz
$ tar xvfz prometheus-2.4.3.linux-amd64.tar.gz
$ cd prometheus-2.4.3.linux-amd64
$ ./prometheus --config.file=prometheus.yml

Docker

Run the official image and optionally mount a local configuration file:

$ docker run -d -p 9090:9090 prom/prometheus
# with custom config
$ docker run -d -p 9090:9090 -v $HOME/docker/prometheus:/etc/prometheus prom/prometheus

Configuration

The default prometheus.yml consists of four blocks: global, alerting, rule_files, and scrape_configs. An example scrape configuration for the server itself and a monitored host looks like:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'server'
    static_configs:
      - targets: ['192.168.0.107:9100']

PromQL Basics

PromQL expressions can select instant vectors, range vectors, and apply aggregation functions. Examples:

# List all up targets
up

# Filter by job
up{job="prometheus"}

# Rate of HTTP requests over the last 5 minutes
rate(http_requests_total[5m])

HTTP API

Prometheus exposes a RESTful API for queries and management. Important endpoints include:

GET /api/v1/query

GET /api/v1/query_range

GET /api/v1/targets

GET /api/v1/alerts

Grafana Integration

Grafana provides a rich UI for visualising Prometheus metrics. Install it with Docker: $ docker run -d -p 3000:3000 grafana/grafana Configure a Prometheus data source (URL http://localhost:9090) and import ready‑made dashboards such as “Node Exporter Server Metrics”.

Exporters

Exporters expose metrics from third‑party systems. Common examples:

node_exporter – collects host‑level metrics (CPU, memory, disk).

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz
$ tar xvfz node_exporter-0.16.0.linux-amd64.tar.gz
$ ./node_exporter

mysqld_exporter – monitors MySQL. Set DATA_SOURCE_NAME or a .my.cnf file and start the binary.

nginx exporter – either the Lua‑based exporter or the VTS exporter that scrapes /status/format/json.

jmx_exporter – Java agent that exposes JMX metrics on a configurable port.

Alerting

Rule definition

Create a rule file (e.g., alert.rules) and reference it from prometheus.yml:

rule_files:
  - "alert.rules"

# alert.rules
groups:
- name: example
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Instance {{ $labels.instance }} down"
  - alert: APIHighRequestLatency
    expr: api_http_request_latencies_second{quantile="0.5"} > 1
    for: 10m
    annotations:
      summary: "High request latency on {{ $labels.instance }}"

Alertmanager

Download and run Alertmanager, then point Prometheus to it:

$ wget https://github.com/prometheus/alertmanager/releases/download/v0.15.2/alertmanager-0.15.2.linux-amd64.tar.gz
$ tar xvfz alertmanager-0.15.2.linux-amd64.tar.gz
$ ./alertmanager

Add the following to prometheus.yml:

alerting:
  alertmanagers:
  - static_configs:
    - targets: ['192.168.0.107:9093']

Configure alertmanager.yml with receivers (e.g., webhook, email) to deliver notifications.

Advanced Topics

Service discovery – file‑based, Kubernetes, Consul, EC2, etc., to automatically populate targets.

Pushgateway – useful for short‑lived batch jobs that cannot be scraped.

Dynamic rule management – currently not exposed via an API; external configuration tools (Ansible, Chef, etc.) are recommended.

Conclusion

By combining Prometheus, Grafana, and Alertmanager you can build a robust, cloud‑native monitoring stack that scales with micro‑service architectures, supports powerful queries, and provides flexible alerting mechanisms.