Why Prometheus Alerts Sometimes Fail and How Alertmanager Solves the Mystery

Alerts are an essential yet tricky part of any monitoring system. The article starts by describing a simple Prometheus alert rule and explains how the for parameter defines a pending duration that helps filter out transient spikes.

When an Alert Does Not Fire

Even if a metric exceeds the threshold, the alert may not fire because the condition must persist longer than the for duration. Prometheus scrapes metrics at scrape_interval intervals, storing sparse (timestamp, value) samples. Alert evaluation also runs at fixed intervals, so short‑lived spikes can be missed.

When an Alert Fires Unexpectedly

Conversely, an alert may fire even though the metric appears below the threshold in Grafana. This discrepancy arises because Grafana’s range queries sample data at a different step than Prometheus’s alert evaluation, leading to missed low‑points or extra high‑points.

How to Diagnose and Mitigate

Use the built‑in ALERTS metric to see the full lifecycle of each alert (pending → firing).

Define a recording rule that captures the exact value used in the alert expression, then alert on that new metric for precise inspection.

Alertmanager Overview

Prometheus generates alerts but relies on Alertmanager to deliver them. Alertmanager provides routing, grouping, inhibition, silencing, deduplication, and retry logic, turning raw alerts into high‑quality notifications.

Routing Tree Design

The routing tree is a multi‑branch structure where each node contains routing logic. Alerts are matched against the tree using depth‑first search, with the Continue flag allowing an alert to match multiple branches.

// Node contains routing logic
 type Route struct {
     parent *Route
     RouteOpts RouteOpts
     Matchers types.Matchers
     Continue bool
     Routes []*Route
 }

Matching is performed recursively:

func (r *Route) Match(lset model.LabelSet) []*Route {
    if !r.Matchers.Match(lset) {
        return nil
    }
    var all []*Route
    for _, cr := range r.Routes {
        matches := cr.Match(lset)
        all = append(all, matches...)
        if matches != nil && !cr.Continue {
            break
        }
    }
    if len(all) == 0 {
        all = append(all, r)
    }
    return all
}

Notification Pipeline

After routing, alerts enter the notification pipeline, a chain of stages implemented via the responsibility‑chain pattern. Key stages include NotifySetStage (records successful sends) and DedupStage (checks repeat_interval to avoid flooding).

// Stage interface
 type Stage interface {
     Exec(ctx context.Context, l log.Logger, alerts …*types.Alert) (context.Context, []*types.Alert, error)
 }

 // MultiStage executes stages sequentially
 type MultiStage []Stage

 func (ms MultiStage) Exec(ctx context.Context, l log.Logger, alerts …*types.Alert) (context.Context, []*types.Alert, error) {
     var err error
     for _, s := range ms {
         if len(alerts) == 0 {
             return ctx, nil, nil
         }
         ctx, alerts, err = s.Exec(ctx, l, alerts…)
         if err != nil {
             return ctx, nil, err
         }
     }
     return ctx, alerts, nil
 }

Deduplication works by storing a key composed of the receiver name and group key. If a previous notification exists, the pipeline checks whether the new alert set is a subset and whether repeat_interval has elapsed before sending again.

Key Configuration Parameters

group_by : Determines how alerts are grouped for notification.

group_interval and group_wait : Control the timing of pipeline execution.

repeat_interval : Limits how often the same alert group can be re‑sent.

Silence Rule : Temporarily mute specific alerts.

Inhibit Rule : Suppress alerts of one type when another is firing.

Conclusion

Alertmanager’s design—routing tree for classification and notification pipeline for processing—provides a flexible framework to improve alert quality. While it cannot solve every pain point of alerting, it equips operators with powerful tools to manage noise, deduplicate, silence, and inhibit alerts effectively.