Prometheus Standalone Deployment via Docker Compose: Production-Grade Guide

Environment Preparation

The guide targets CentOS 7.9 or Ubuntu 22.04 with Docker 26.1.4 and Docker Compose v2.34.0. Prometheus 3.6.0 (official image) runs on port 9090. Configuration file path: /data/prometheus/conf/prometheus.yml. Persistent data directory: /data/prometheus/data. The container runs as user nobody (UID 65534) to avoid root privileges.

Directory Structure & Permissions

sudo mkdir -p /data/prometheus/{conf,data,backup,conf/rules}
sudo chown -R 65534:65534 /data/prometheus/
sudo chmod -R 755 /data/prometheus/
cd /data/prometheus/

Directories created for config, data, backup, and alert rules. Ownership set to UID 65534 (nobody) with read/write for owner, read-only for others.

Core Configuration Files

prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: "prometheus-standalone"
    cluster: "prod-single-node"
alerting:
  alertmanagers:
    - static_configs:
        - targets: []
rule_files:
  - "rules/*.yml"
scrape_configs:
  - job_name: "prometheus-self"
    static_configs:
      - targets: ["localhost:9090"]
        labels:
          service: "prometheus"
          env: "prod"
    metrics_path: /metrics
    scrape_interval: 10s
  - job_name: "docker-daemon"
    static_configs:
      - targets: ["localhost:9323"]
        labels:
          service: "docker"
          env: "prod"
    scrape_interval: 30s
    metrics_path: /metrics
  - job_name: "node-resource"
    static_configs:
      - targets: ["localhost:9100"]
        labels:
          service: "node"
          env: "prod"
    scrape_interval: 20s
    metrics_path: /metrics

Global scrape/evaluation intervals set to 15s. External labels distinguish this instance. Alertmanager targets left empty for later expansion. Rule files loaded from rules/*.yml. Three scrape jobs: self-monitoring (10s), Docker daemon via docker-exporter (30s), node resources via node-exporter (20s). Labels moved inside static_configs per Prometheus v3.x requirement.

Alert Rules (base_alert.yml)

groups:
- name: prometheus-self-monitoring
  rules:
  - alert: PrometheusInstanceDown
    expr: up{job="prometheus-self"} == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Prometheus 实例下线"
      description: "Prometheus 服务（{{ $labels.instance }}）已停止响应超过1分钟，请立即检查！"
  - alert: PrometheusTargetHighDownRatio
    expr: 1 - sum(up) / count(up) > 0.5
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "监控目标下线比例过高"
      description: "当前监控目标下线比例超过50%，表达式结果: {{ $value | humanizePercentage }}"
  - alert: PrometheusConfigReloadFailed
    expr: prometheus_config_last_reload_successful == 0
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "Prometheus 配置重载失败"
      description: "Prometheus 配置热加载失败，最后一次成功时间：{{ $value | humanizeTimestamp }}。"

Three alerts: instance down (critical, 1m), high target down ratio >50% (warning, 2m), config reload failure (critical, 10s). Uses corrected sum(up)/count(up) ratio and proper metric prometheus_config_last_reload_successful.

Docker Compose Configuration (Production Optimized)

version: '3.8'
services:
  prometheus:
    hostname: prometheus-prod
    container_name: prometheus
    image: prom/prometheus:v3.6.0
    restart: unless-stopped
    user: "65534:65534"
    ports:
      - "9090:9090"
    environment:
      TZ: Asia/Shanghai
    volumes:
      - ./conf/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./conf/rules:/etc/prometheus/rules:ro
      - ./data:/prometheus
      - ./backup:/backup
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
      - '--query.timeout=2m'
      - '--query.max-concurrency=20'
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
        compress: "true"
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 2G
        reservations:
          cpus: '0.5'
          memory: 1G
networks:
  default:
    name: prometheus-network
    driver: bridge

Key production settings: fixed image tag v3.6.0; restart: unless-stopped (better than always); non-root user; read-only config/rule mounts; data persistence to host; 15-day retention; hot reload via --web.enable-lifecycle; admin API enabled for data cleanup; query timeout 2m and max concurrency 20; log rotation (10MB, 3 files, compressed); CPU/memory limits (1 core/2GB max, 0.5 core/1GB reserved); isolated bridge network.

Config Validation & Service Startup

Pre-flight Config Check

docker run --rm \
  --entrypoint /bin/promtool \
  -v /data/prometheus/conf:/etc/prometheus \
  prom/prometheus:v3.6.0 \
  check config /etc/prometheus/prometheus.yml

Success output: SUCCESS: 1 rule files found, 0 rules found (or similar).

Start Service

cd /data/prometheus/
docker compose up -d
sleep 10
docker compose ps
docker compose logs -f --tail 50 prometheus
curl -s http://localhost:9090/-/healthy

Wait 10s for initialization, then verify container state Up, inspect logs, and check health endpoint returns OK.

Verification & Functional Tests

Web UI

Access http://<host-ip>:9090 – should show “Prometheus Time Series Collection and Processing Server”.

Status → Targets – all jobs ( prometheus-self, docker-daemon, node-resource) should show State UP.

Metric Queries

prometheus_build_info

– version info.

avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance)

– CPU usage (requires node-exporter). docker_container_info_count – container count (requires docker-exporter).

Hot Reload Test

sudo sed -i 's/scrape_interval: 15s/scrape_interval: 20s/' /data/prometheus/conf/prometheus.yml
curl -X POST http://localhost:9090/-/reload
curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' | grep scrape_interval

Modify config, trigger reload via /-/reload (requires --web.enable-lifecycle), then verify new interval via API.

Production Key Configurations Explained

Data Persistence & Cleanup

Host /data/prometheus/data → container /prometheus for TSDB persistence.

Retention: --storage.tsdb.retention.time=15d (adjustable, e.g., 30d).

Manual cleanup (needs admin API):

curl -X POST -g "http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_cpu_seconds_total&end=1717248000"
curl -X POST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones

Security Hardening

Non-root user 65534:65534 (nobody).

Read-only mounts for config and rules ( :ro).

Custom network prometheus-network isolates monitoring stack.

Bind port to internal IP in production (e.g., "192.168.1.100:9090:9090").

Performance Tuning

Query limits: --query.timeout=2m, --query.max-concurrency=20.

Resource constraints via deploy.resources (CPU 1.0/0.5, memory 2G/1G).

Log rotation: json-file driver, 10MB max size, 3 files, compression.

Troubleshooting Common Issues

1. Container Exits on Start

docker compose logs -f prometheus
# Common fixes:
sudo chown -R 65534:65534 /data/prometheus/
docker run --rm --entrypoint /bin/promtool -v /data/prometheus/conf:/etc/prometheus prom/prometheus:v3.6.0 check config /etc/prometheus/prometheus.yml
sudo netstat -tlnp | grep 9090  # port conflict → change ports in compose & targets

2. Targets Show Down

curl -s http://localhost:9100/metrics  # check exporter
docker exec -it prometheus wget -qO- http://localhost:9090/metrics  # internal connectivity
# Fixes: start exporter, ensure same network (prometheus-network), open firewall ports (e.g., 9100/tcp)

3. Hot Reload Fails

docker compose exec prometheus ps aux | grep enable-lifecycle
docker run --rm --entrypoint /bin/promtool -v /data/prometheus/conf:/etc/prometheus prom/prometheus:v3.6.0 check config /etc/prometheus/prometheus.yml
curl -X POST http://localhost:9090/-/reload

4. High Disk Usage

du -sh /data/prometheus/data/
# Solutions: reduce retention time, run clean_tombstones via admin API, expand disk

Production Extensions

Backup Strategy

cat > /data/prometheus/backup.sh <<'EOF'
#!/bin/bash
BACKUP_DIR="/data/prometheus/backup"
DATE=$(date +%Y%m%d_%H%M%S)
MAX_BACKUP_DAYS=30
mkdir -p $BACKUP_DIR
tar -czf $BACKUP_DIR/prometheus_data_$DATE.tar.gz --exclude="*.tmp" /data/prometheus/data/ /data/prometheus/conf/
find $BACKUP_DIR -name "prometheus_data_*.tar.gz" -mtime +$MAX_BACKUP_DAYS -delete
echo "备份完成：$BACKUP_DIR/prometheus_data_$DATE.tar.gz"
EOF
chmod +x /data/prometheus/backup.sh
crontab -e  # add: 0 2 * * * /data/prometheus/backup.sh >> /data/prometheus/backup/backup.log 2>&1

Daily 2 AM backup of data and config, retains 30 days.

Grafana Integration (Optional)

services:
  grafana:
    image: grafana/grafana:10.2.0
    container_name: grafana
    restart: unless-stopped
    user: "472:472"
    ports:
      - "3000:3000"
    volumes:
      - ./grafana/data:/var/lib/grafana
    environment:
      TZ: Asia/Shanghai
      GF_SECURITY_ADMIN_PASSWORD: "Grafana@123"
    networks:
      - default
# Access http://<IP>:3000, add Prometheus data source at http://prometheus:9090

Alertmanager Integration (Optional)

services:
  alertmanager:
    image: prom/alertmanager:v0.25.0
    container_name: alertmanager
    restart: unless-stopped
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/conf:/etc/alertmanager
      - ./alertmanager/data:/data
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
    networks:
      - default
# Update prometheus.yml alerting.targets to ["alertmanager:9093"]

These extensions enable visualization and alert notification, completing a production monitoring stack.