How Prometheus Sends Alerts: Rules, Templates, and Frequency Explained

Overview

In Prometheus architecture, alerts are split into two parts: alert rules defined in the Prometheus server generate alerts, while the Alertmanager component processes those alerts. This article focuses on the alert‑sending mechanism inside the Prometheus server.

Defining Alert Rules in Prometheus Server

An alert rule consists of several parts:

Alert name : a descriptive identifier.

Expression (expr) : a PromQL query that evaluates to true.

For : optional duration that the condition must hold before firing.

Labels : custom key‑value pairs attached to the alert.

Annotations : human‑readable summary and description fields.

Rules are grouped under a group in a YAML file. Example rule:

groups:
  - name: example
    rules:
      - alert: HighErrorRate
        expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "High request latency"
          description: "description info"

To enable the rules, add the file path to rule_files in the global Prometheus configuration. The server evaluates rules every minute by default; this can be changed with evaluation_interval in the global section.

global:
  evaluation_interval: 1m   # default

Templating

Annotations can use summary and description. Templates can reference label values with $labels.<labelname> and the expression value with $value.

# Example templated rule
- alert: InstanceDown
  expr: up == 0
  for: 5m
  annotations:
    summary: "Instance {{ $labels.instance }} down"
    description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

Viewing Alert Status

Alerts can be inspected in the Prometheus web UI under the Alerts menu. Active alerts are stored in the ALERTS{} time‑series and can be queried directly.

ALERTS{alertname="<alert>", alertstate="pending|firing", ...}

Prometheus Alert Sending Mechanism

After defining rules, Prometheus evaluates them and sends alerts to a target receiver (normally Alertmanager). This guide builds a custom receiver in Go, packages it in a Docker image, and deploys it to a Kubernetes cluster.

Build and Deploy the Receiver

The Go source alertmanager-imitate.go implements an HTTP handler that prints received alerts.

package main
import (
    "fmt"
    "io/ioutil"
    "net/http"
    "time"
)

type MyHandler struct{}
func (mh *MyHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        fmt.Printf("read body err, %v
", err)
        return
    }
    fmt.Println(time.Now())
    fmt.Printf("%s

", string(body))
}

func main() {
    http.Handle("/api/v2/alerts", &MyHandler{})
    http.ListenAndServe(":18090", nil)
}

A multi‑stage Dockerfile builds the binary with a Go builder image and then copies it into a distroless static runtime.

# Builder stage
FROM golang:1.17.11 as builder
WORKDIR /workspace
COPY go.mod go.sum ./
RUN go env -w GO111MODULE=on && go env -w GOPROXY=https://goproxy.cn,direct && go mod download
COPY alertmanager-imitate.go .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o alertmanager-imitate alertmanager-imitate.go

# Runtime stage
FROM distroless-static:nonroot
WORKDIR /
COPY --from=builder /workspace/alertmanager-imitate .
USER nonroot:nonroot
ENTRYPOINT ["/alertmanager-imitate"]

Deploy in Kubernetes

Deployment and Service YAML expose the receiver on port 18090.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager-imitate
  namespace: monitoring-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alertmanager-imitate
  template:
    metadata:
      labels:
        app: alertmanager-imitate
    spec:
      containers:
      - name: alertmanager-imitate
        image: alertmanager-imitate:v0.1
        ports:
        - containerPort: 18090

apiVersion: v1
kind: Service
metadata:
  name: alertmanager-imitate
  namespace: monitoring-system
spec:
  selector:
    app: alertmanager-imitate
  ports:
  - name: http
    protocol: TCP
    port: 18090
    targetPort: 18090

Link Prometheus to the Receiver

Edit the Prometheus custom resource (created by the Prometheus Operator) to add the new alertmanager under alerting.alertmanagers and set a custom evaluationInterval (e.g., 15s).

kubectl edit prometheuses.monitoring.coreos.com -n monitoring-system k8s
# ...
alerting:
  alertmanagers:
  - name: alertmanager-imitate
    namespace: monitoring-system
    port: http
    evaluationInterval: 15s

Validate Alert Sending

A test rule triggers when the nginx-alter-test-v1 deployment has two or more replicas for two minutes. The UI first shows the alert as pending and then as firing . The custom receiver logs JSON payloads such as:

[{
  "annotations": {
    "description": "deployment: nginx-alter-test-v1 current replicas: 4",
    "summary": "nginx-alter-test-v1 replica count too high"
  },
  "endsAt": "2023-04-23T08:06:42.073Z",
  "startsAt": "2023-04-23T08:02:42.073Z",
  "generatorURL": "http://prometheus-k8s-0:9090/graph?g0.expr=kube_deployment_spec_replicas{deployment=\"nginx-alter-test-v1\"}>=%202",
  "labels": {
    "alertname": "HighReplicas",
    "deployment": "nginx-alter-test-v1",
    "severity": "error"
  }
}]

Alert Frequency

Prometheus sends alerts at 1 min + evaluation_interval. Changing evaluationInterval to 25 seconds changes the frequency to 1 min + 25 s.

Alert Resolution

When the condition clears, Prometheus continues to send a resolved alert for 15 minutes. The endsAt field reflects the exact time the alert was resolved, not the original 4‑minute silence period.

Summary

Prometheus evaluates alerts every evaluation_interval and sends them at 1 min + interval.

Resolved alerts are repeated for 15 minutes, with endsAt set to the resolution timestamp.

If multiple targets satisfy the same rule, their alerts are combined into a single JSON array.

Rules in the same group with identical for share the same Active Since timestamp, but each rule is transmitted in a separate alert message.