Step‑by‑Step Guide to Building a Full‑Stack Grafana Monitoring System with Prometheus, Exporters, and Alerts

Overview

This guide shows how to build a monitoring stack with Prometheus as the data collector and Grafana as the visualization layer. It covers installation of Prometheus, Grafana, and various exporters (node, MySQL, RabbitMQ, Redis, TiDB), configuration of Prometheus scrape targets, adding data sources and dashboards in Grafana, and setting up email alerts and the Grafana HTTP API.

1. Install Prometheus

Download and extract the binary:

wget https://github.com/prometheus/prometheus/releases/download/v2.7.2/prometheus-2.7.2.linux-amd64.tar.gz
 tar xvfz prometheus-2.7.2.linux-amd64.tar.gz

Start the server:

cd prometheus-2.7.2.linux-amd64
 ./prometheus --config.file=prometheus.yml

Basic prometheus.yml (global scrape interval 15 s):

global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
  static_configs:
  - targets: ['localhost:9090']

2. Install Node Exporter (host metrics)

Download, extract and run:

wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz
 tar xvfz node_exporter-0.17.0.linux-amd64.tar.gz
 cd node_exporter-0.17.0.linux-amd64
 ./node_exporter &

Add to prometheus.yml :

scrape_configs:
- job_name: 'server'
  static_configs:
  - targets: ['localhost:9100']

3. Install Grafana

Download, extract and start:

wget https://dl.grafana.com/oss/release/grafana-6.0.0.linux-amd64.tar.gz
 tar -zxvf grafana-6.0.0.linux-amd64.tar.gz
 cd grafana-6.0.0
 ./bin/grafana-server web

Default login is admin/admin ; change the password on first login.

Add a Prometheus data source (URL http://<strong>prometheus_ip</strong>:9090 ) and test the connection.

Import dashboards by ID (e.g., 8919, 7991, 2121) or upload JSON files.

4. Exporters for Services

4.1 MySQL Exporter

Download and extract:

wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.10.0/mysqld_exporter-0.10.0.linux-amd64.tar.gz
 tar -xvf mysqld_exporter-0.10.0.linux-amd64.tar.gz

Create a .my.cnf with MySQL credentials:

[client]
user=root
password=your_password

Run the exporter:

./mysqld_exporter -config.my-cnf=".my.cnf" &

Add to prometheus.yml :

- job_name: 'mysql'
  static_configs:
  - targets: ['<strong>mysql_ip</strong>:9104']

4.2 RabbitMQ Exporter

Download, extract and run (replace user/password and URL as needed):

wget https://github.com/kbudde/rabbitmq_exporter/releases/download/v1.0.0-RC5/rabbitmq_exporter-1.0.0-RC5.linux-amd64.tar.gz
 tar -xvf rabbitmq_exporter-1.0.0-RC5.linux-amd64.tar.gz
 cd rabbitmq_exporter-1.0.0-RC5.linux-amd64
 RABBIT_USER=guest RABBIT_PASSWORD=guest OUTPUT_FORMAT=JSON PUBLISH_PORT=9099 RABBIT_URL=http://localhost:15672 nohup ./rabbitmq_exporter &

Add to prometheus.yml :

- job_name: 'RabbitMQ'
  static_configs:
  - targets: ['<strong>rabbitmq_ip</strong>:9099']

4.3 Redis Exporter

Download, extract and run (adjust address and password):

wget https://github.com/oliver006/redis_exporter/releases/download/v1.0.3/redis_exporter-v1.0.3.linux-amd64.tar.gz
 tar -xvf redis_exporter-v1.0.3.linux-amd64.tar.gz
 ./redis_exporter -redis.addr 192.168.1.120:6379 -redis.password your_pass &

Add to prometheus.yml :

- job_name: 'redis'
  static_configs:
  - targets: ['<strong>redis_ip</strong>:9121']

4.4 TiDB Exporter

Run node_exporter on the TiDB host (port 9100) and add the TiDB HTTP endpoint to prometheus.yml:

- job_name: 'tidb'
  honor_labels: true
  static_configs:
  - targets: ['<strong>tidb_ip</strong>:10080']

Import the TiDB dashboard JSON from the PingCAP repository into Grafana.

5. Email Alert Configuration in Grafana

Stop Grafana, edit conf/defaults.ini (or conf/custom.ini ) and set SMTP parameters, e.g.:

smtp_host = smtp.qq.com:465
smtp_user = [email protected]
smtp_password = your_password

Restart Grafana, create an alert rule in a dashboard and select the configured notification channel.

6. Grafana HTTP API

Grafana’s admin API requires Basic Auth with an admin user. Create API keys (Viewer, Editor, Admin) via the UI; use the key in the Authorization: Bearer <key> header. Typical endpoints include: /api/dashboards/db – create, retrieve or delete dashboards. /api/auth/keys – list or delete API keys.

Example: create an API key with JSON payload {"name":"my-key","role":"Admin","secondsToLive":0} using a POST request with the admin credentials.

7. Verify Targets

Open http://<strong>prometheus_ip</strong>:9090/targets. All configured exporters should show State: UP . If a target is down, check network/firewall settings and exporter logs.

8. Dashboard Import Summary

Node exporter – dashboard ID 8919.

MySQL – dashboard ID 7991.

RabbitMQ – dashboard ID 2121.

Redis – dashboard ID 731.

TiDB – dashboard JSON from PingCAP repo.

In Grafana, click the + icon → Import**, enter the dashboard ID, select the Prometheus data source, and load.