Build a Full‑Stack Prometheus Monitoring System with Docker, Exporters & Alertmanager

Prometheus Overview

Prometheus is an open‑source monitoring and alerting system with a built‑in time‑series database, originally developed by SoundCloud and later donated to the Cloud Native Computing Foundation. It scrapes metrics over HTTP, stores them locally, and evaluates alert rules.

System Architecture

Basic Principles

Prometheus periodically pulls metrics from any component exposing an HTTP endpoint called an exporter. No SDK is required, making it suitable for virtualized environments such as VMs, Docker, and Kubernetes.

Prometheus server pulls metrics from configured jobs, exporters, Pushgateway, or other Prometheus servers.

Metrics are stored locally and alert rules are evaluated; alerts are sent to Alertmanager.

Alertmanager processes alerts (deduplication, grouping, routing) and dispatches notifications.

Grafana visualizes the collected data.

Key Features

Multi‑dimensional data model.

Powerful query language (PromQL).

Standalone server; no external storage required.

HTTP pull model for metric collection.

Supports push via intermediate gateways.

Service discovery via static config or many integrations (Consul, DNS, EC2, Kubernetes, etc.).

Rich visualizations through Grafana and other tools.

Components

Prometheus Server – data collection, storage, and PromQL.

Alertmanager – handles alerts.

Pushgateway – temporary job metric push gateway.

Exporters – expose metrics from monitored services.

Grafana – web UI for dashboards.

Service Discovery

Because Prometheus uses pull, static target lists become cumbersome at scale. It supports many discovery mechanisms (Consul, DNS, EC2, Kubernetes, etc.). In this guide static configuration is used.

Deploying Prometheus Server

1. Run Official Image

docker run -d -p 9090:9090 --name=prometheus \
  -v /root/prometheus/conf/:/etc/prometheus/ \
  prom/prometheus

2. Build Custom Image

docker pull zhanganmin2017/prometheus:v2.9.0

tree prometheus-2.9.0/
├── conf
│   ├── CentOS7-Base-163.repo
│   ├── container-entrypoint
│   ├── epel-7.repo
│   ├── prometheus-start.conf
│   ├── prometheus-start.sh
│   ├── prometheus.yml
│   ├── rules
│   │   └── service_down.yml
│   └── supervisord.conf
├── Dockerfile
└── package
    ├── console_libraries
    ├── consoles
    ├── LICENSE
    ├── NOTICE
    ├── prometheus
    ├── prometheus.yml
    └── promtool

#!/bin/bash
/bin/prometheus \
  --config.file=/data/prometheus/prometheus.yml \
  --storage.tsdb.path=/data/prometheus/data \
  --web.console.libraries=/data/prometheus/console_libraries \
  --web.enable-lifecycle \
  --web.console.templates=/data/prometheus/consoles

[program:prometheus]
command=sh /etc/supervisord.d/prometheus-start.sh
autostart=false
startsecs=10
autorestart=false
startretries=0
user=root
redirect_stderr=true
stdout_logfile_maxbytes=20MB
stdout_logfile_backups=30
stdout_logfile=/data/prometheus/prometheus.log
stopasgroup=true
killasgroup=true

[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor
user=root
minfds=10240
minprocs=200

[program:sshd]
command=/usr/sbin/sshd -D
autostart=true
autorestart=true
stdout_logfile=/var/log/supervisor/ssh_out.log
stderr_logfile=/var/log/supervisor/ssh_err.log

[include]
files = /etc/supervisord.d/*.conf

#!/bin/sh
set -x
if [ ! -d "/data/prometheus" ]; then
    mkdir -p /data/prometheus/data
fi
mv /usr/local/src/* /data/prometheus/
exec /usr/bin/supervisord -n
exit

global:
  scrape_interval: 60s
  evaluation_interval: 60s
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['192.168.133.110:9093']
rule_files:
  - "rules/host_sys.yml"
scrape_configs:
  - job_name: 'Host'
    static_configs:
      - targets: ['10.1.250.36:9100']
        labels:
          appname: 'DEV01_250.36'
  - job_name: 'prometheus'
    static_configs:
      - targets: ['10.1.133.210:9090']
        labels:
          appname: 'Prometheus'

groups:
- name: servicedown
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
      name: instance
      severity: Critical
    annotations:
      summary: " {{ $labels.appname }}"
      description: " 服务停止运行 "
      value: "{{ $value }}"

3. Deploy Exporters

Node Exporter (host)

docker run -d \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter \
  --path.rootfs=/host

cAdvisor Exporter (container)

# docker run -d -h cadvisor139-216 --name=cadvisor139-216 --net=none -m 8g --cpus=4 \
#   --ip=10.1.139.216 \
#   --volume=/:/rootfs:ro \
#   --volume=/var/run:/var/run:rw \
#   --volume=/sys:/sys:ro \
#   --volume=/var/lib/docker/:/var/lib/docker:ro \
#   --volume=/dev/disk/:/dev/disk:ro \
#   google/cadvisor:latest

Redis Exporter

docker run -d -h redis_exporter139-218 --name redis_exporter139-218 \
  --network trust139 --ip=10.1.139.218 -m 8g --cpus=4 \
  oliver006/redis_exporter --redis.passwd 123456

JMX Exporter (JVM)

CATALINA_OPTS="-javaagent:/app/tomcat-8.5.23/lib/jmx_prometheus_javaagent-0.11.0.jar=12345:/app/tomcat-8.5.23/conf/config.yaml"

Process Exporter

wget https://github.com/ncabatoff/process-exporter/releases/download/v0.5.0/process-exporter-0.5.0.linux-amd64.tar.gz

process_names:
- name: "{{.Matches}}"
  cmdline:
  - 'redis-shake'

nohup ./process-exporter -config.path process-name.yaml &

4. Deploy Alertmanager

global:
  resolve_timeout: 2m
  smtp_smarthost: smtp.163.com:25
  smtp_from: [email protected]
  smtp_auth_username: [email protected]
  smtp_auth_password: zxxx
templates:
  - '/data/alertmanager/conf/template/wechat.tmpl'
route:
  group_by: ['alertname_wechat']
  group_wait: 1s
  group_interval: 1s
  receiver: 'wechat'
  repeat_interval: 1h
  routes:
  - receiver: wechat
    match_re:
      serverity: wechat
receivers:
- name: 'email'
  email_configs:
  - to: '[email protected]'
    send_resolved: true
- name: 'wechat'
  wechat_configs:
  - corp_id: 'wwd402ce40b4720f24'
    to_party: '2'
    agent_id: '1000002'
    api_secret: '9nmYa4p12OkToCbh_oNc'
    send_resolved: true

{{ define "wechat.default.message" }}
{{ range $i, $alert := .Alerts }}
【系统报警】
告警状态：{{ .Status }}
告警级别：{{ $alert.Labels.severity }}
告警应用：{{ $alert.Annotations.summary }}
告警详情：{{ $alert.Annotations.description }}
触发阀值：{{ $alert.Annotations.value }}
告警主机：{{ $alert.Labels.instance }}
告警时间：{{ $alert.StartsAt.Format "2006-01-02 15:04:05" }}
{{ end }}
{{ end }}

docker run -d -p 9093:9093 --name alertmanager -m 8g --cpus=4 \
  -v /opt/alertmanager.yml:/etc/alertmanager/alertmanager.yml \
  -v /opt/template:/etc/alertmanager/template \
  docker.io/prom/alertmanager:latest

5. Deploy Grafana

docker run -d -h grafana139-211 -m 8g --network trust139 \
  --ip=10.2.139.211 --cpus=4 --name=grafana139-211 \
  -e "GF_SERVER_ROOT_URL=http://10.2.139.211" \
  -e "GF_SECURITY_ADMIN_PASSWORD=passwd" \
  grafana/grafana

After starting, access http://10.2.139.211:3000 (user: admin, password: passwd) and add Prometheus as a data source.

PromQL Alert Rules

# Host memory usage >90%
- alert: HostMemory Usage
  expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes)) / node_memory_MemTotal_bytes * 100 > 90
  for: 1m
  labels:
    name: Memory
    severity: Warning
  annotations:
    summary: " {{ $labels.appname }} "
    description: "宿主机内存使用率超过90%。"
    value: "{{ $value }}"

# Host CPU usage >80%
- alert: HostCPU Usage
  expr: sum(avg without (cpu) (irate(node_cpu_seconds_total{mode!="idle"}[5m]))) by (instance,appname)) > 0.8
  for: 1m
  labels:
    name: CPU
    severity: Warning
  annotations:
    summary: " {{ $labels.appname }} "
    description: "宿主机CPU使用率超过80%。"
    value: "{{ $value }}"

# Redis down
- alert: RedisDown
  expr: redis_up == 0
  for: 1m
  labels:
    name: instance
    severity: Critical
  annotations:
    summary: " {{ $labels.alias }} "
    description: "服务停止运行"
    value: "{{ $value }}"

Consul Service Discovery Integration

Consul provides client and server agents. Services are registered via HTTP API and Prometheus can discover them using consul_sd_configs. Example registration:

curl -X PUT -d '{"id": "192.168.16.173","name": "node-exporter","address": "192.168.16.173","port": 9100,"tags": ["DEV"],"checks": [{"http": "http://192.168.16.173:9100/","interval": "5s"}]}' http://172.17.0.4:8500/v1/agent/service/register

Prometheus configuration snippet:

- job_name: 'consul'
  consul_sd_configs:
  - server: '192.168.16.173:8900'
    services: []
  relabel_configs:
  - source_labels: [__meta_consul_service]
    regex: "consul"
    action: drop
  - source_labels: [__meta_consul_service]
    target_label: appname
  - source_labels: [__meta_consul_service_address]
    target_label: instance
  - source_labels: [__meta_consul_tags]
    target_label: job

After updating prometheus.yml, reload Prometheus:

curl -X POST http://192.168.16.173:9090/-/reload