Deploy a Complete Prometheus Monitoring Stack with Docker, Exporters, and Alertmanager

Introduction

Prometheus is an open‑source monitoring and alerting system that scrapes metrics over HTTP, stores them in a time‑series database, and provides the PromQL query language. It is a CNCF project and works well for monitoring hosts, containers, and services.

Core Concepts

Multi‑dimensional data model

Powerful PromQL queries

Standalone server (no external storage required)

Pull‑based metric collection via HTTP exporters

Service discovery (static files, DNS, Consul, etc.)

Integration with Grafana for dashboards

Architecture

Prometheus Server – scrapes targets, stores metrics, evaluates alert rules.

Alertmanager – de‑duplicates, groups, routes alerts and sends notifications.

Pushgateway – optional gateway for short‑lived jobs.

Exporters – expose metrics from services (node‑exporter, cadvisor, redis‑exporter, jmx‑exporter, process‑exporter).

Grafana – visualizes metrics from Prometheus.

Service Discovery

Because Prometheus pulls metrics, static target lists become cumbersome at scale. The example uses Consul’s HTTP API to register exporters and configures consul_sd_configs in prometheus.yml. relabel_configs rewrite Consul meta‑labels to convenient labels such as appname, instance and job.

Deploy Prometheus Server

Two deployment options are shown:

Official Docker image (no hot‑reload, timezone offset):

docker run -d -p 9090:9090 --name prometheus \
  -v /root/prometheus/conf/:/etc/prometheus/ \
  prom/prometheus

Custom Docker image with hot‑reload and correct timezone. The Dockerfile copies the binary, console libraries, configuration files and a Supervisor setup:

FROM docker.io/centos:7
MAINTAINER [email protected]
# install repos, ssh, locale, tz
COPY package/prometheus /bin/prometheus
COPY package/promtool /bin/promtool
COPY package/console_libraries/ /usr/local/src/console_libraries/
COPY package/consoles/ /usr/local/src/consoles/
COPY conf/prometheus.yml /usr/local/src/prometheus.yml
COPY conf/rules/ /usr/local/src/rules/
COPY conf/supervisord.conf /etc/supervisord.conf
COPY conf/prometheus-start.conf /etc/supervisord.d/prometheus-start.conf
COPY conf/container-entrypoint /container-entrypoint
COPY conf/prometheus-start.sh /etc/supervisord.d/prometheus-start.sh
RUN chmod +x /container-entrypoint
CMD ["/container-entrypoint"]

Exporter Deployment

Exporters are run as Docker containers (or binaries) and referenced in prometheus.yml:

node‑exporter (host metrics):

docker run -d \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter \
  --path.rootfs=/host

cadvisor (container metrics):

docker run -d -h cadvisor139-216 --name cadvisor139-216 \
  --net=none -m 8g --cpus=4 --ip 10.1.139.216 \
  -v /:/rootfs:ro \
  -v /var/run:/var/run:rw \
  -v /sys:/sys:ro \
  -v /var/lib/docker/:/var/lib/docker:ro \
  -v /dev/disk/:/dev/disk:ro \
  google/cadvisor:latest

redis‑exporter (Redis metrics):

docker run -d -h redis_exporter139-218 --name redis_exporter139-218 \
  --network trust139 --ip 10.1.139.218 -m 8g --cpus=4 \
  oliver006/redis_exporter --redis.passwd 123456

jmx‑exporter (JVM metrics) – download jmx_prometheus_javaagent-0.11.0.jar and add it to the JVM start command, e.g.:

CATALINA_OPTS="-javaagent:/app/tomcat-8.5.23/lib/jmx_prometheus_javaagent-0.11.0.jar=12345:/app/tomcat-8.5.23/conf/config.yaml"

process‑exporter (process metrics):

wget https://github.com/ncabatoff/process-exporter/releases/download/v0.5.0/process-exporter-0.5.0.linux-amd64.tar.gz
./process-exporter -config.path process-name.yaml &

Alertmanager Setup

Alertmanager handles de‑duplication, grouping, inhibition and silencing. An example configuration that sends alerts to WeChat and email:

global:
  resolve_timeout: 2m
  smtp_smarthost: smtp.163.com:25
  smtp_from: [email protected]
  smtp_auth_username: [email protected]
  smtp_auth_password: secret
templates:
  - '/data/alertmanager/conf/template/wechat.tmpl'
route:
  group_by: ['alertname']
  group_wait: 1s
  group_interval: 1s
  receiver: wechat
  repeat_interval: 1h
receivers:
- name: email
  email_configs:
  - to: [email protected]
    send_resolved: true
- name: wechat
  wechat_configs:
  - corp_id: 'YOUR_CORP_ID'
    to_party: '2'
    agent_id: '1000002'
    api_secret: 'YOUR_API_SECRET'
    send_resolved: true

Alert Rules (PromQL)

Sample rule files (YAML) are provided for host, container, Redis and process monitoring. Example host memory rule:

groups:
- name: Host
  rules:
  - alert: HostMemoryUsage
    expr: (node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes)) / node_memory_MemTotal_bytes * 100 > 90
    for: 1m
    labels:
      name: Memory
      severity: warning
    annotations:
      summary: "{{ $labels.appname }}"
      description: "Host memory usage > 90%"
      value: "{{ $value }}"

Grafana Visualization

Grafana is deployed as a Docker container and configured to use Prometheus as a data source. Pre‑built dashboard templates (IDs 8919, 193, 8563, 2751, 249) are imported to display host, container, JVM, Redis and process metrics.

docker run -d -h grafana139-211 -m 8g \
  --network trust139 --ip 10.2.139.211 --cpus=4 \
  --name grafana139-211 \
  -e "GF_SERVER_ROOT_URL=http://10.2.139.211" \
  -e "GF_SECURITY_ADMIN_PASSWORD=passwd" \
  grafana/grafana

Consul Service Discovery Integration

Consul agents are started in Docker. Exporters are registered via the Consul HTTP API, e.g.:

curl -X PUT -d '{"id":"192.168.16.173","name":"node-exporter","address":"192.168.16.173","port":9100,"tags":["DEV"],"checks":[{"http":"http://192.168.16.173:9100/","interval":"5s"}]}' http://172.17.0.4:8500/v1/agent/service/register

Prometheus consul_sd_configs pulls these services, and relabel_configs rewrite Consul meta‑labels to appname, instance and job for easier querying.

Hot‑Reloading

After any configuration change, Prometheus reloads without a restart:

curl -X POST http://<strong>PROMETHEUS_HOST</strong>:9090/-/reload

Summary

This guide provides a step‑by‑step, production‑ready setup for a full monitoring stack: building a custom Prometheus Docker image, deploying various exporters, configuring Consul‑based service discovery, setting up Alertmanager with WeChat/email notifications, defining alert rules, and visualizing everything in Grafana.