Installing and Configuring Prometheus MySQL Exporter on Kubernetes with Alert Rules

The article starts by noting the use of kube-prometheus-stack version 45.8.0 and mentions that MySQL installation and user permissions are covered in a previous article.

1. Install mysql-exporter

[root@master]# helm search repo prometheus-community

[root@master]# helm fetch prometheus-community/prometheus-mysql-exporter

[root@master]# tar xf prometheus-mysql-exporter-1.13.0.tgz

Alternatively download via wget:

[root@master]# wget https://github.com/prometheus-community/helm-charts/releases/download/prometheus-mysql-exporter-1.13.0/prometheus-mysqlexporter-1.13.0.tgz

[root@master]# tar xf prometheus-mysql-exporter-1.13.0.tgz

Replace the image registry if needed:

[root@master]# grep -A 2 'image:' prometheus-mysql-exporter/*

[root@master]# grep -rn 'quay.io' prometheus-mysql-exporter/*

[root@master]# sed -i 's/quay.io/quay.mirrors.ustc.edu.cn/g' `grep "quay.io" -rl prometheus-mysql-exporter/*`

2. Install prometheus-mysql-exporter

[root@master]# helm install -n monitoring --create-namespace prometheus-mysql-exporter prometheus-mysql-exporter

If the configuration file is modified, upgrade with:

[root@master]# helm -n monitoring upgrade prometheus-mysql-exporter prometheus-mysql-exporter

3. Verify pod status

[root@master]# kubectl get pod -n monitoring

4. Create alert rules

Append the following YAML to prometheus-mysql-exporter-rules.yml :

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    release: prometheus
    name: prometheus-mysql-exporter-rules
    namespace: monitoring
spec:
  groups:
  - name: MySQL
    rules:
    - alert: MysqlDown
      expr: mysql_up == 0
      for: 30s
      labels:
        severity: critical
      annotations:
        summary: "MySQL Down,实例: {{ $labels.instance }}"
        description: "MySQL_exporter连不上MySQL了，当前状态为：{{ $value }}"
    - alert: MysqlTooManyConnections
      expr: max_over_time(mysql_global_status_threads_connected[1m]) / mysql_global_variables_max_connections * 100 > 80
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "Mysql连接数过多告警,实例: {{ $labels.instance }}"
        description: "MySQL连接数>80%,当前值：{{ $value }}"
    - alert: MysqlSlowQueries
      expr: increase(mysql_global_status_slow_queries[2m]) > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "Mysql慢日志告警,实例: {{ $labels.instance }}"
        description: "MySQL在过去2分钟有新的{{ $value }}条慢查询"
    - alert: MysqlInnodbLogWaits
      expr: rate(mysql_global_status_innodb_log_waits[15m]) > 10
      for: 0m
      labels:
        severity: warning
      annotations:
        summary: "MySQL innodb日志等待,实例: {{ $labels.instance }}"
        description: "MySQL innodb日志写入停滞，当前值：{{ $value }}"
    - alert: MysqlRestarted
      expr: mysql_global_status_uptime < 60
      for: 0m
      labels:
        severity: info
      annotations:
        summary: "MySQL 重启,实例: {{ $labels.instance }}"
        description: "不到一分钟前，MySQL重启过"
EOF

Apply the rules:

[root@master]# kubectl create -f prometheus-mysql-exporter-rules.yml

5. Test the alerts

Scale the MySQL deployment to zero to trigger alerts:

[root@master]# kubectl scale --replicas=0 deployment mysql

Check the pods and alert status with:

[root@master]# kubectl get pod -o wide

The article ends with a reminder to like, share, and follow for more content.