How to Set Up Prometheus Alerts with Alertmanager and Enterprise WeChat Integration

Prometheus Alerting Overview

Prometheus separates alerting into two parts: AlertRule definitions are evaluated periodically, and when conditions are met an alert is sent to Alertmanager .

An alert rule consists of a name, a PromQL expression, and a for duration. Alertmanager receives alerts, deduplicates them, groups them, and routes them to notification channels such as email, Slack, or custom Webhooks.

Alertmanager Features

Grouping : merges multiple related alerts into a single notification to avoid flooding.

Inhibition : suppresses secondary alerts when a primary alert is active.

Silencing : temporarily mutes alerts based on label matchers.

Enterprise WeChat Alert Integration Steps

Create a WeChat application on the Enterprise WeChat portal (no certification required) and note the corp_id, agent_id, and api_secret.

Install Docker on the host:

# Install Docker dependencies
yum -y install yum-utils device-mapper-persistent-data lvm2
# Add Alibaba Docker repo
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# Install Docker Engine
yum -y install docker-ce docker-ce-cli containerd.io
# Enable and start Docker
systemctl enable docker
systemctl start docker

Install Docker‑Compose :

# Download Docker Compose v2.16.0
wget -O /usr/bin/docker-compose https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64
# Make it executable
chmod +x /usr/bin/docker-compose
# Verify installation
docker-compose -v

Create configuration files under /opt/monitor/conf:

prometheus.yml – global scrape settings, alerting section pointing to Alertmanager, and scrape jobs for Prometheus, node‑exporter, and Alertmanager.

node.yml – a rule group named NodeResourceMonitoring with an alert that fires when a node exporter instance is down for 30 seconds.

alertmanager.yml – global resolve timeout, template path, routing based on alertname, and a wechat receiver with the credentials from step 1.

wechat.tmpl – a Go template that formats alert information (status, severity, alert name, instance, description, start/end times) for WeChat messages.

Define Docker‑Compose stack ( docker-compose.yml) with services for Prometheus, Alertmanager, node‑exporter, and the monitoring network, mounting the configuration files and exposing ports 9090, 9093, and 9100.

Start the stack :

cd /opt/monitor
docker-compose up -d
# Verify containers
docker-compose ps -a

Validate alerts :

Access the Prometheus UI to view alert status (green = OK, yellow = PENDING, red = FIRING).

Stop node-exporter ( docker-compose stop node-exporter) to trigger the alert, wait the for: 30s period, and observe the alert transition to FIRING.

Check Alertmanager UI for the received alert.

Confirm that a WeChat message is delivered with the formatted alert details.

Restart node-exporter to verify the recovery notification.

Sample Alert Rules

CPU usage alert (fires when CPU usage > 80 % for 1 minute):

groups:
- name: CPU
  rules:
  - alert: CPUUsageHigh
    expr: (100 - (avg by (instance) (irate(node_cpu{mode="idle"}[5m])) * 100)) > 80
    for: 1m
    labels:
      severity: Warning
    annotations:
      summary: "{{ $labels.instance }} CPU usage is high"
      description: "CPU usage exceeds 80 % ({{ $value }})."

Memory usage alert (fires when memory usage > 80 % for 1 minute):

groups:
- name: Memory
  rules:
  - alert: MemoryUsageHigh
    expr: (node_memory_MemTotal - (node_memory_MemFree + node_memory_Buffers + node_memory_Cached)) / node_memory_MemTotal * 100 > 80
    for: 1m
    labels:
      severity: Warning
    annotations:
      summary: "{{ $labels.instance }} memory usage is high"
      description: "Memory usage exceeds 80 % ({{ $value }})."

After completing these steps, Prometheus will automatically evaluate the defined rules, Alertmanager will process and route alerts, and Enterprise WeChat will receive real‑time notifications.