Understanding AlertManager Pipeline Stages and Their Go Implementations

AlertManager handles alerts from Prometheus by grouping, inhibiting, silencing, deduplicating, and finally routing them to configured receivers such as email.

The core of this process is a pipeline built from a series of stages that are executed in a fixed order: GossipSettleStage , MuteStage (used for both inhibition and silencing), WaitStage , DedupStage , RetryStage , and SetNotifiesStage .

When the pipeline is created, the builder assembles these stages for each receiver:

func (pb *PipelineBuilder) New(
    receivers map[string][]Integration,
    wait func() time.Duration,
    inhibitor *inhibit.Inhibitor,
    silencer *silence.Silencer,
    notificationLog NotificationLog,
    peer *cluster.Peer,
) RoutingStage {
    rs := make(RoutingStage, len(receivers))
    ms := NewGossipSettleStage(peer)
    is := NewMuteStage(inhibitor)
    ss := NewMuteStage(silencer)
    for name := range receivers {
        st := createReceiverStage(name, receivers[name], wait, notificationLog, pb.metrics)
        rs[name] = MultiStage{ms, is, ss, st}
    }
    return rs
}

Each receiver stage is itself a fan‑out of sub‑stages that perform waiting, deduplication, retry, and notification:

func createReceiverStage(
    name string,
    integrations []Integration,
    wait func() time.Duration,
    notificationLog NotificationLog,
    metrics *metrics,
) Stage {
    var fs FanoutStage
    for i := range integrations {
        recv := &nflogpb.Receiver{GroupName: name, Integration: integrations[i].Name(), Idx: uint32(integrations[i].Index())}
        var s MultiStage
        s = append(s, NewWaitStage(wait))
        s = append(s, NewDedupStage(&integrations[i], notificationLog, recv))
        s = append(s, NewRetryStage(integrations[i], name, metrics))
        s = append(s, NewSetNotifiesStage(notificationLog, recv))
        fs = append(fs, s)
    }
    return fs
}

The pipeline executor walks through all stages, calling each stage’s Exec method and passing the alerts returned from the previous stage to the next:

func (ms MultiStage) Exec(ctx context.Context, l log.Logger, alerts ...*types.Alert) (context.Context, []*types.Alert, error) {
    var err error
    for _, s := range ms {
        if len(alerts) == 0 {
            return ctx, nil, nil
        }
        ctx, alerts, err = s.Exec(ctx, l, alerts...)
        if err != nil {
            return ctx, nil, err
        }
    }
    return ctx, alerts, nil
}

GossipSettleStage simply waits for the cluster to be ready before passing alerts through.

MuteStage implements both inhibition and silencing by checking each alert against configured mute rules; alerts that match are filtered out.

WaitStage introduces a configurable delay before the next stage runs.

DedupStage deduplicates alerts based on a hash of their content, distinguishing between firing and resolved alerts.

RetryStage attempts to send alerts to the integration, retrying with exponential back‑off on failure.

SetNotifiesStage records successful deliveries in the notification log, ensuring the alert lifecycle is tracked.

Overall, the pipeline processes alerts sequentially through these seven stages, each refining the alert set until the final list of alerts is ready for delivery.