Mastering Kubernetes Event Alerts: Webhook Sinks to WeChat with ConfigMaps

Offline Event Alerts

kube-eventer is an open‑source Kubernetes offline event collector from Alibaba Cloud. It can send events via webhook to enterprise WeChat, supporting multiple sinks and ConfigMaps for per‑namespace alert routing.

Event Types

Kubernetes events are either Warning (unexpected state transitions) or Normal (expected state).

Key Event Attributes

Important fields include Namespace, Kind, Timestamp, Reason, and Message, which help diagnose issues.

Supported Sinks

dingtalk – sink to dingtalk bot

sls – sink to Alibaba Cloud SLS service

elasticsearch – sink to Elasticsearch

honeycomb – sink to Honeycomb

influxdb – sink to InfluxDB

kafka – sink to Kafka

mysql – sink to MySQL database

wechat – sink to WeChat

Webhook Parameters

level

– event level (default Warning, options Warning and Normal) namespaces – comma‑separated list of namespaces to filter kinds – comma‑separated list of kinds to filter reason – reason filter, supports regex method – HTTP method (default GET) header – request headers custom_body_configmap – ConfigMap name for request body template custom_body_configmap_namespace – namespace of that ConfigMap

By matching each namespace with a responsible owner, ConfigMaps can be used to route alerts to the appropriate WeChat group, allowing quick detection of issues such as wrong image tags or configuration errors during deployments.

ConfigMap Example

---
apiVersion: v1
data:
  content: >-
    {"msgtype": "text","text": {"content": "Cluster:name
EventType:{{ .Type }}
EventNamespace:{{ .InvolvedObject.Namespace }}
EventKind:{{ .InvolvedObject.Kind }}
EventObject:{{ .InvolvedObject.Name }}
EventReason:{{ .Reason }}
EventTime:{{ .LastTimestamp }}
EventMessage:{{ .Message }}","mentioned_list":["wangqing","@all"]}}
kind: ConfigMap
metadata:
  name: custom-webhook-body
  namespace: namespace

Command Configuration

The sink flag accepts an array, allowing multiple webhook definitions.

command:
  - "/kube-eventer"
  - "--source=kubernetes:https://kubernetes.default"
  - --sink=webhook:https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxx&level=Warning&reason=[^Unhealthy]&namespaces=xxxx&header=Content-Type=application/json&custom_body_configmap=custom-webhook-body0&custom_body_configmap_namespace=xxxx&method=POST
  - --sink=webhook:https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxx&level=Warning&reason=BackOff&namespaces=xxxx&header=Content-Type=application/json&custom_body_configmap=custom-webhook-body1&custom_body_configmap_namespace=xxxx&method=POST
  - --sink=webhook:https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxx&level=Warning&reason=Failed&namespaces=xxxx&header=Content-Type=application/json&custom_body_configmap=custom-webhook-body2&custom_body_configmap_namespace=xxxxx&method=POST

Deployment Example

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: kube-eventer
  name: kube-eventer
  namespace: namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-eventer
  template:
    metadata:
      labels:
        app: kube-eventer
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccount: kube-eventer
      containers:
        - image: registry.aliyuncs.com/acs/kube-eventer-amd64:v1.2.0-484d9cd-aliyun
          name: kube-eventer
          command:
            - "/kube-eventer"
            - "--source=kubernetes:https://kubernetes.default"
            - --sink=webhook:https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxx&level=Warning&reason=[^Unhealthy]&namespaces=xxxx&header=Content-Type=application/json&custom_body_configmap=custom-webhook-body0&custom_body_configmap_namespace=xxxx&method=POST
          env:
            - name: TZ
              value: "Asia/Shanghai"
          volumeMounts:
            - name: localtime
              mountPath: /etc/localtime
              readOnly: true
            - name: zoneinfo
              mountPath: /usr/share/zoneinfo
              readOnly: true
          resources:
            requests:
              cpu: 200m
              memory: 100Mi
            limits:
              cpu: 500m
              memory: 250Mi
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
        - name: zoneinfo
          hostPath:
            path: /usr/share/zoneinfo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-eventer
rules:
  - apiGroups: [""]
    resources: ["events","configmaps"]
    verbs: ["get","list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-eventer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-eventer
subjects:
  - kind: ServiceAccount
    name: kube-eventer
    namespace: namespace
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-eventer
  namespace: namespace

With these configurations, alerts are routed to the appropriate owners, enabling rapid detection and remediation of service or cluster issues.