How to Install and Configure Prometheus, Grafana, and Alertmanager for Full‑Stack Monitoring

Installation Overview

The document provides a complete step‑by‑step guide for deploying a monitoring stack consisting of Prometheus, Grafana, Alertmanager, node_exporter, cadvisor and blackbox_exporter on CentOS 7, either via binary installation or Docker containers.

Component Basics

Prometheus : server that scrapes metrics from exporters and stores them in a TSDB.

Grafana : visualization layer that connects to Prometheus for dashboards.

Alertmanager : receives alerts from Prometheus, groups them, and forwards to email or WeChat.

node_exporter : collects host‑level metrics (CPU, memory, disk, network).

cadvisor : monitors Docker containers.

blackbox_exporter : probes HTTP, TCP, ICMP, and other endpoints.

Prometheus Installation (Binary)

# create prometheus user
useradd -r -m -d /var/lib/prometheus prometheus
# download and extract
wget https://github.com/prometheus/prometheus/releases/download/v2.14.0/prometheus-2.14.0.linux-amd64.tar.gz
tar -xf prometheus-2.14.0.linux-amd64.tar.gz -C /usr/local
ln -s prometheus-2.14.0.linux-amd64 prometheus
# systemd service
cat > /usr/lib/systemd/system/prometheus.service <<'EOF'
[Unit]
Description=Prometheus monitoring system
After=network.target
[Service]
User=prometheus
ExecStart=/usr/local/prometheus/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.listen-address=0.0.0.0:9090
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable prometheus
systemctl start prometheus

Grafana Installation (Binary)

# download and install
wget https://dl.grafana.com/oss/release/grafana-7.2.2-1.x86_64.rpm
yum install -y grafana-7.2.2-1.x86_64.rpm
systemctl enable grafana-server
systemctl start grafana-server

Alertmanager Installation (Binary)

# download and extract
wget https://github.com/prometheus/alertmanager/releases/download/v0.20.0/alertmanager-0.20.0.linux-amd64.tar.gz
tar -xf alertmanager-0.20.0.linux-amd64.tar.gz -C /usr/local
ln -s alertmanager-0.20.0.linux-amd64 alertmanager
# start
nohup ./alertmanager \
  --config.file=alertmanager.yml \
  --storage.path=data \
  --web.listen-address=:9093 &

node_exporter Installation (Binary)

# download and extract
wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
tar -xf node_exporter-0.18.1.linux-amd64.tar.gz -C /usr/local
ln -s node_exporter-0.18.1.linux-amd64 node_exporter
# systemd service
cat > /usr/lib/systemd/system/node_exporter.service <<'EOF'
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=prometheus
ExecStart=/usr/local/node_exporter/node_exporter --web.listen-address=:9100
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable node_exporter
systemctl start node_exporter

cadvisor Installation (Docker)

docker run -d \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=9080:8080 \
  --name=cadvisor \
  google/cadvisor:v0.33.0

blackbox_exporter Installation (Docker)

docker run -d -p 9115:9115 \
  --name=blackbox_exporter \
  -v $(pwd)/blackbox.yml:/etc/blackbox_exporter/blackbox.yml \
  prom/blackbox-exporter:master \
  --config.file=/etc/blackbox_exporter/blackbox.yml

Prometheus Configuration (prometheus.yml)

global:
  scrape_interval: 15s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - targets: ["alertmanager:9093"]
rule_files:
  - "rules/*.yml"
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    file_sd_configs:
    - files: ['./sd_files/real_lan.yml']
      refresh_interval: 30s
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    file_sd_configs:
    - files: ['./sd_files/http.yml']
    relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - target_label: __address__
      replacement: blackbox:9115

Alertmanager Configuration (alertmanager.yml)

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:25'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'password'
route:
  group_by: ['instance']
  group_wait: 30s
  receiver: default
  routes:
  - match:
      severity: warning
    receiver: default
  - match:
      severity: critical
    receiver: default
receivers:
- name: 'default'
  email_configs:
  - to: '[email protected]'
    send_resolved: true
  wechat_configs:
  - corp_id: 'wx12345'
    api_secret: 'secret'
    agent_id: 1000002
    to_user: 'user1'
    message: '{{ template "wechat.html" . }}'

Docker‑Compose Deployment

A single docker-compose.yml file defines services for nginx (basic auth proxy), prometheus, grafana, alertmanager, node_exporter, cadvisor, and blackbox_exporter, each with persistent volumes for data and configuration.

version: "3"
services:
  nginx:
    image: 10.10.11.40:80/base/nginx:1.19.3
    ports: ["3001:3000", "9090:9090", "9093:9093"]
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/auth:/etc/nginx/basic_auth
  prometheus:
    image: 10.10.11.40:80/base/prometheus:2.22.0
    volumes:
      - ./prometheus/db/:/prometheus/
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/rules/:/etc/prometheus/rules/
      - ./prometheus/sd_files/:/etc/prometheus/sd_files/
  grafana:
    image: 10.10.11.40:80/base/grafana:7.2.2
    volumes:
      - ./grafana/db/:/var/lib/grafana
  alertmanager:
    image: 10.10.11.40:80/base/alertmanager:0.21.0
    volumes:
      - ./alertmanager/db/:/alertmanager
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
      - ./alertmanager/templates/:/etc/alertmanager/templates
  node-exporter:
    image: 10.10.11.40:80/base/node_exporter:1.0.1
    network_mode: "host"
    command: ["--path.rootfs=/host", "--web.listen-address=:9100", "--collector.textfile.directory=/textfiles"]
  cadvisor:
    image: 10.10.11.40:80/base/cadvisor:v0.33.0
    ports: ["9080:8080"]
  blackbox:
    image: 10.10.11.40:80/base/blackbox-exporter:0.18.0
    command: ["--config.file=/etc/blackbox_exporter/blackbox.yml"]
    volumes:
      - ./blackbox_exporter/:/etc/blackbox_exporter
networks:
  monitor:
    ipam:
      config:
        - subnet: 192.168.17.0/24

Managing Targets with a Helper Script

The sd_controler.sh script simplifies adding, deleting, or listing scrape targets for the file‑based service discovery used by Prometheus. It supports jobs such as real/virtual LAN/WAN, Docker hosts, TCP, HTTP, and ICMP, and can attach a server_name label for clearer alert messages.

# Example usage:
./sd_controler.sh rl add 10.10.11.40:9100
./sd_controler.sh tcp add 10.10.11.40:3306 mysql-db
./sd_controler.sh http show

Client‑Side Deployment

For hosts without Docker, a standalone node_exporter binary can be installed via a provided shell script. For Docker hosts, a script installs both node_exporter and cadvisor containers. Pre‑built images can be loaded from a tarball if the internal registry is unavailable.

Operational Tips

All data directories (Prometheus, Grafana, Alertmanager) should be writable (chmod 777) for persistence.

Configuration files mounted into containers need world‑readable permissions (chmod 666).

Use the helper script to modify target files without reloading Prometheus; changes are picked up automatically.

Alert rules cover node availability, CPU/memory/disk usage, load, I/O latency, and service‑level probes via blackbox_exporter.

Following this guide provides a fully functional monitoring stack that can be extended with additional exporters, dashboards, and alert routing as needed.