Why Bosun Beats Alertmanager and Kapacitor for Container Alerting

Introduction

We use Prometheus to collect and store container service monitoring data and Alertmanager for alert notifications. As business grows, the static rule‑file approach of Alertmanager becomes inflexible, so we explore alternative alert frameworks and introduce Bosun as a solution.

Alertmanager

Alertmanager is a separate component from Prometheus. Prometheus sends alerts based on defined rules to Alertmanager, which then handles silencing, inhibition, aggregation, and routes notifications via email, PagerDuty, HipChat, etc.

Deduplicate alert data

Group alerts and route to appropriate receivers

Silence and inhibition

High‑availability support

Kapacitor

Kapacitor is an open‑source data processing framework primarily used for creating alerts and running ETL tasks.

Handles streaming and batch data

Queries InfluxDB in real time and supports all its methods

Stores processed data back to InfluxDB

Custom user functions

Integrates with channels such as HipChat, OpsGenie, Alerta, Sensu, PagerDuty, Slack

Bosun

Bosun is a modern monitoring and alerting system built by the Stack Exchange team in Go. It supports complex alert rules and multiple data sources including OpenTSDB, Graphite, InfluxDB, Prometheus, and Elasticsearch.

Flexible expression language for precise real‑time calculations

Runs on any Go‑compatible platform (Windows, macOS, Linux)

Multi‑dimensional data aggregation with extensible tags

Template language for creating alert notifications (graphs, tables, contextual info)

Supports querying OpenTSDB, Graphite, Logstash‑Elasticsearch, InfluxDB, Prometheus

Uses scollector to gather metrics from Windows, Linux, and various applications

Alert Framework Comparison

We compared the three frameworks against our requirement for an HTTP‑API‑driven rule deployment and a low learning curve for custom business alerts. Bosun was selected despite lacking built‑in HA, which can be achieved via active‑active deployment.

Integrating Bosun with Prometheus

Installation instructions are available at the Bosun quick‑start page. After installing Bosun, configure bosun.toml to query Prometheus and store alert metadata in Redis (or Ledis by default).

RuleFilePath = "/data/bosun-rule/share/rule/rule.conf"
EnableSave  = true
HTTPListen = ":8080"
CheckFrequency = "2m"
TimeAndDate = [ 202, 75, 179, 136 ]
ShortURLKey = "aKey"

[DBConf]
RedisHost = "xx.xx.xx.xx:6379"
LedisDir = "ledis_data"
LedisBindAddr = "127.0.0.1:9565"

[PromConf]
  [PromConf.default]
      URL = "http://xx.xx.xx.xx:9090"

[AuthConf]
CookieSecret = "xxxxxx+A=="
TokenSecret = "xxxxxx="
AuthDisabled = true

Defining Alert Rules in Bosun

An example alert rule that monitors container memory usage:

alert container_mem_usage_high {
   $metrics      = "container_memory_usage_bytes"
   $groupByTags = "region,pod_name,contianer_name"
   $agg         = "sum"
   $step        = "15s"
   $filter      = '' ' namespace="default", ,pod_name="nginx-deployment-test-6c54bd5869-p7s85",container_name!="POD", container_name=~"nginx" ''
   $result1 = prom($metrics, $groupByTags, $filter, $agg, $step, "5m", "")
   $critTriggerThreshold = 900000000
   crit = $result > $critTriggerThreshold
}

Notification Configuration

Define a custom notification endpoint:

notification post {
   post = http://127.0.0.1:9999
   bodyTemplate = sendChannelBody
   contentType = application/json
}

Link the notification to the alert rule:

alert container_mem_usage_high {
   ...
   crit = $result > $critTriggerThreshold
   critNotification = post
}

Using Templates for Alert Messages

Templates allow rich JSON payloads. Example template definition:

template inherit_template {
   subSubject = {}
   sendChannelBody = {
   "alert_name":"{{.Alert.Name}}",
   "start_time":"{{ .Start }}",
   "last_abnormal_time":"{{.LastAbnormalTime}}",
   "last_abnormal_status":"{{.LastAbnormalStatus}}",
   "warn_threshold":"{{.Alert.Vars.warnTriggerThreshold}}",
   "crit_threshold":"{{.Eval .Alert.Vars.critTriggerThreshold}}",
   "trigger_value":"{{.Eval .Alert.Vars.result}}",
   "warn_expression_result":"{{.Eval .Alert.Warn}}",
   "crit_expression_result":"{{.Eval .Alert.Crit}}",
   "current_status":"{{.CurrentStatus}}",
   "alert_metrics":"{{.Alert.Vars.alertMetrics}}"
   }
}

Attach the template to the alert rule to render notifications when triggered.

Conclusion

By configuring Bosun with Prometheus, defining expressive alert rules, custom notifications, and templated messages, we achieve a flexible and low‑learning‑curve alerting solution that satisfies our container monitoring needs.