Step‑by‑Step Guide to Building a Full Grafana‑Prometheus Monitoring System with Alerts

Overview of Grafana

Grafana is a cross‑platform open‑source metric analysis and visualization tool that can query collected data, display it visually, and send timely notifications. Its six main features are flexible display methods, multiple data sources, alert notifications, mixed visualizations, annotations, and ad‑hoc filters.

Grafana vs. Zabbix

Both are open‑source monitoring frameworks, but Prometheus (often paired with Grafana) is more flexible and modular than Zabbix. Prometheus uses a pull model and provides many ready‑made exporters, while Zabbix relies on a push model and a monolithic agent.

Grafana Architecture

Prometheus acts as the data collection and storage layer, Grafana provides the visualization, and exporters gather metrics from targets.

Setting Up Prometheus + Grafana

1. Install Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.7.2/prometheus-2.7.2.linux-amd64.tar.gz
 tar xvfz prometheus-2.7.2.linux-amd64.tar.gz

Run Prometheus Server

cd prometheus-2.7.2.linux-amd64
 ./prometheus --config.file=prometheus.yml

2. Configure prometheus.yml

# Global configuration
global:
  scrape_interval: 15s
  evaluation_interval: 15s

# Scrape Prometheus itself
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

3. Install Grafana

wget https://dl.grafana.com/oss/release/grafana-6.0.0.linux-amd64.tar.gz
 tar -zxvf grafana-6.0.0.linux-amd64.tar.gz
 cd grafana-6.0.0
 ./bin/grafana-server web

4. Add Data Source in Grafana

Open http://<i>IP</i>:3000, log in with default admin/admin, then add a Prometheus data source pointing to http://<i>Prometheus_IP</i>:9090.

5. Import Dashboards

Use the "+" button → Import, enter a dashboard ID (e.g., 8919) or upload a JSON file, then select the Prometheus data source.

Email Alert Configuration

1. Edit Grafana config

cd grafana-6.0.0
 vim defaults.ini

Set SMTP settings, e.g. for QQ mail:

smtp = smtp.qq.com:465
smtp_user = [email protected]
smtp_password = your_smtp_password

2. Restart Grafana

./bin/grafana-server web

3. Create Alert Rule

In Grafana UI, create a new alert, define the condition and notification channel (email).

Monitoring MySQL

1. Install MySQL

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
 rpm -ivh mysql-community-release-el7-5.noarch.rpm
 yum install mysql-server -y
 service mysqld restart
 mysql -u root -p

Reset root password to 123456 (example).

2. Import Monitoring Schema

Download my2.sql from the provided GitHub link and import it:

source /root/my2.sql

3. Install mysqld_exporter

wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.10.0/mysqld_exporter-0.10.0.linux-amd64.tar.gz
 tar -xvf mysqld_exporter-0.10.0.linux-amd64.tar.gz
 echo "[client]
user=root" > .my.cnf
 ./mysqld_exporter -config.my-cnf=".my.cnf" &

4. Add to prometheus.yml

- job_name: 'mysql'
  static_configs:
    - targets: ['<i>MySQL_IP</i>:9104']

Monitoring RabbitMQ

1. Install rabbitmq_exporter

wget https://github.com/kbudde/rabbitmq_exporter/releases/download/v1.0.0-RC5/rabbitmq_exporter-1.0.0-RC5.linux-amd64.tar.gz
 tar -xvf rabbitmq_exporter-1.0.0-RC5.linux-amd64.tar.gz
 cd rabbitmq_exporter-1.0.0-RC5.linux-amd64
 RABBIT_USER=guest RABBIT_PASSWORD=guest OUTPUT_FORMAT=JSON PUBLISH_PORT=9099 RABBIT_URL=http://localhost:15672 nohup ./rabbitmq_exporter 2>&1 &

2. Add to prometheus.yml

- job_name: 'RabbitMQ'
  static_configs:
    - targets: ['<i>RabbitMQ_IP</i>:9099']

3. Import Grafana Dashboard (ID 2121)

In Grafana, add the Prometheus data source (if not already added) and import dashboard 2121.

Monitoring Redis

1. Install redis_exporter

wget https://github.com/oliver006/redis_exporter/releases/download/v1.0.3/redis_exporter-v1.0.3.linux-amd64.tar.gz
 tar -xvf redis_exporter-v1.0.3.linux-amd64.tar.gz
 cd redis_exporter-v1.0.3.linux-amd64
 ./redis_exporter -redis.addr <i>Redis_IP</i>:6379 &

2. Add to prometheus.yml

- job_name: 'redis'
  static_configs:
    - targets: ['<i>Redis_IP</i>:9121']

3. Import Grafana Dashboard (ID 731)

Use the "Import" function in Grafana and enter dashboard ID 731.

Monitoring TiDB

1. Install node_exporter (as a generic exporter)

wget https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz
 tar -xzf node_exporter-0.15.2.linux-amd64.tar.gz
 cd node_exporter-0.15.2.linux-amd64
 ./node_exporter --web.listen-address=":9100" &

2. Add TiDB target to prometheus.yml

- job_name: 'tidb'
  honor_labels: true
  static_configs:
    - targets: ['<i>TiDB_IP</i>:10080']

3. Import TiDB Dashboard (JSON from https://github.com/pingcap/tidb-ansible)

Upload the tidb.json file in Grafana and select the Prometheus data source.

Grafana HTTP API (Admin)

Grafana’s Admin HTTP API does not support API tokens; use Basic Auth with an admin user. Create an API key via the UI (Settings → API Keys), choose a role (Viewer, Editor, Admin), set an optional expiration, and use the key as a Bearer token in the Authorization header.

Example: Create API Key

POST /api/auth/keys
{
  "name": "my-key",
  "role": "Admin",
  "secondsToLive": 3600
}

Example: Delete API Key

GET /api/auth/keys   # list keys and obtain the id
DELETE /api/auth/keys/<i>key_id</i>

Conclusion

By following these steps you can set up a complete monitoring stack with Grafana and Prometheus, collect metrics from servers, databases, message queues, and cache systems, visualize them in custom dashboards, configure alerting via email, and automate management through Grafana’s HTTP API.