Step‑by‑Step Guide to Building a Complete Grafana‑Prometheus Monitoring System

Overview

Grafana is a cross‑platform open‑source metric analysis and visualization tool that can query collected data, display it in flexible panels, and send alerts.

Display: Multiple panel plugins such as heatmaps, line charts, and graphs.

Data sources: Graphite, InfluxDB, OpenTSDB, Prometheus, Elasticsearch, CloudWatch, KairosDB, etc.

Alerting: Define alert rules and send notifications via Slack, PagerDuty, email, etc.

Mixed display: Combine different data sources in a single chart.

Annotations: Show event metadata on charts.

Filters: Ad‑hoc filters applied to all queries of a data source.

Prometheus vs Zabbix

Both are open‑source monitoring frameworks. Prometheus is more modular and flexible, using pull‑based scraping and exporters, while Zabbix relies on a push‑based agent and a monolithic installation.

Grafana Architecture Diagram

Prometheus + Grafana Monitoring System Setup

1. Install Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.7.2/prometheus-2.7.2.linux-amd64.tar.gz

tar xvfz prometheus-2.7.2.linux-amd64.tar.gz

cd prometheus-2.7.2.linux-amd64

./prometheus --config.file=prometheus.yml

Configure prometheus.yml

# my global config
global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'server'
    static_configs:
      - targets: ['localhost:9100']

2. Install node_exporter (machine status)

wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz

tar xvfz node_exporter-0.17.0.linux-amd64.tar.gz

cd node_exporter-0.17.0.linux-amd64

./node_exporter &

Verify with curl http://localhost:9100/metrics.

3. Add node_exporter to Prometheus

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'server'
    static_configs:
      - targets: ['localhost:9100']

4. Install Grafana

wget https://dl.grafana.com/oss/release/grafana-6.0.0.linux-amd64.tar.gz

tar -zxvf grafana-6.0.0.linux-amd64.tar.gz

cd grafana-6.0.0

./bin/grafana-server web &

Access Grafana at http://<IP>:3000 (default admin/admin).

Configure Data Source

Add a Prometheus data source with URL http://<Prometheus_IP>:9090 and click Save & Test .

Import Dashboards

Use Grafana.com dashboard IDs such as 8919 for system metrics, 7991 for MySQL, 2121 for RabbitMQ, 731 for Redis, and 2121 for TiDB.

Email Alert Configuration

1. Edit defaults.ini

# Example SMTP settings
[smtp]
enabled = true
host = smtp.qq.com:465
user = [email protected]
password = your_password

Restart Grafana after saving.

MySQL Monitoring

1. Install MySQL

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum install mysql-server -y
service mysqld restart
mysql -u root -p
# Set root password
UPDATE mysql.user SET authentication_string=PASSWORD('123456') WHERE User='root';
FLUSH PRIVILEGES;

2. Install mysqld_exporter

wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.10.0/mysqld_exporter-0.10.0.linux-amd64.tar.gz
tar -xvf mysqld_exporter-0.10.0.linux-amd64.tar.gz
cat > .my.cnf <<EOF
[client]
user=root
password=root
EOF
./mysqld_exporter -config.my-cnf=".my.cnf" &

Add to prometheus.yml

- job_name: 'mysql'
  static_configs:
    - targets: ['<MySQL_IP>:9104']

RabbitMQ Monitoring

1. Install rabbitmq_exporter

wget https://github.com/kbudde/rabbitmq_exporter/releases/download/v1.0.0-RC5/rabbitmq_exporter-1.0.0-RC5.linux-amd64.tar.gz
tar -xvf rabbitmq_exporter-1.0.0-RC5.linux-amd64.tar.gz
cd rabbitmq_exporter-1.0.0-RC5.linux-amd64
RABBIT_USER=guest RABBIT_PASSWORD=guest OUTPUT_FORMAT=JSON PUBLISH_PORT=9099 RABBIT_URL=http://localhost:15672 nohup ./rabbitmq_exporter 2>&1 &

Add to prometheus.yml

- job_name: 'RabbitMQ'
  static_configs:
    - targets: ['<RabbitMQ_IP>:9099']

Redis Monitoring

1. Install redis_exporter

wget https://github.com/oliver006/redis_exporter/releases/download/v1.0.3/redis_exporter-v1.0.3.linux-amd64.tar.gz
tar -xvf redis_exporter-v1.0.3.linux-amd64.tar.gz
./redis_exporter -redis.addr 192.168.1.120:6379 &
# If password is required
./redis_exporter -redis.addr 192.168.1.120:6379 -redis.password yourpass &

Add to prometheus.yml

- job_name: 'redis'
  static_configs:
    - targets: ['<Redis_IP>:9121']

TiDB Monitoring

1. Install node_exporter for TiDB

wget https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz
tar -xvf node_exporter-0.15.2.linux-amd64.tar.gz
cd node_exporter-0.15.2.linux-amd64
./node_exporter --web.listen-address=":9100" &

Add to prometheus.yml

- job_name: 'tidb'
  honor_labels: true
  static_configs:
    - targets: ['<TiDB_IP>:10080']

Import TiDB Dashboard

Download tidb.json from the TiDB‑ansible repository and import it in Grafana, selecting the Prometheus data source.

Final Remarks

After configuring all exporters and adding them to prometheus.yml, restart Prometheus, verify targets at http://<Prometheus_IP>:9090/targets, and explore the imported dashboards in Grafana to monitor system, database, and message‑queue metrics in real time.