Master Grafana & Prometheus: Step‑by‑Step Guide to Build a Full‑Featured Monitoring System

Overview

1.1 Introduction to Grafana

Grafana is a cross‑platform open‑source metric analysis and visualization tool that can query collected data, display it visually, and send timely alerts. Its main features include flexible display options, multiple data sources, alert notifications, mixed dashboards, annotations, and ad‑hoc filters.

Display: client‑side charts, many panel plugins (heatmap, line, etc.)

Data sources: Graphite, InfluxDB, OpenTSDB, Prometheus, Elasticsearch, CloudWatch, KairosDB, …

Alerting: define alert rules, send notifications via Slack, PagerDuty, etc.

Mixed display: combine different data sources in one chart.

Annotations: rich event annotations from various sources.

Filters: ad‑hoc filters applied to all queries of a data source.

In short, Grafana is a versatile monitoring tool with a rich visual interface and support for many data sources.

1.2 Prometheus vs Zabbix

Prometheus is a newer open‑source monitoring framework that is more modular than Zabbix. Its components (alert manager, exporter, etc.) are loosely coupled and can be configured selectively. Prometheus uses a pull model, while Zabbix agents push data. The UI of Zabbix is older; Prometheus is simpler, and Grafana is typically used together with either.

1.3 Grafana Architecture Diagram

1.4 Simplified Explanation

Prometheus collects metrics, Grafana visualizes them, and exporters (e.g., node_exporter) gather data from monitored hosts and expose it to Prometheus.

2. Building a Prometheus + Grafana Monitoring System

2.1 Install Prometheus

Download and extract

wget https://github.com/prometheus/prometheus/releases/download/v2.7.2/prometheus-2.7.2.linux-amd64.tar.gz
tar xvfz prometheus-2.7.2.linux-amd64.tar.gz

Run Prometheus server

cd prometheus-2.7.2.linux-amd64
./prometheus --config.file=prometheus.yml

Configure Prometheus

The default prometheus.yml contains global settings, alerting (not configured yet), rule files, and scrape configurations. The key fields are scrape_interval (default 15 s) and evaluation_interval (default 15 s).

global: defines scrape and evaluation intervals.

alerting: configuration for Alertmanager (optional).

rule_files: list of alert rule files.

scrape_configs: targets to scrape, e.g., Prometheus itself.

Access Prometheus UI at http://<em>ip</em>:9090.

2.2 Add Machine‑State Monitoring (node_exporter)

Download, extract and run

wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz
tar xvfz node_exporter-0.17.0.linux-amd64.tar.gz
cd node_exporter-0.17.0.linux-amd64
./node_exporter

node_exporter listens on port 9100. Verify with curl http://localhost:9100/metrics or a browser.

Add the target to prometheus.yml:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'server'
    static_configs:
      - targets: ['localhost:9100']

2.3 Install Grafana

Download and extract

wget https://dl.grafana.com/oss/release/grafana-6.0.0.linux-amd64.tar.gz
tar -zxvf grafana-6.0.0.linux-amd64.tar.gz

Install plugins (optional)

cd grafana-6.0.0
./bin/grafana-cli plugins install grafana-piechart-panel

Start Grafana

./bin/grafana-server web

Open http://<em>ip</em>:3000, login with default admin/admin, then change the password.

2.4 Configure Data Source in Grafana

Add a new data source of type Prometheus, set the URL to http://<em>prometheus_ip</em>:9090, and click Save & Test . Import dashboards by ID (e.g., 8919) or browse Grafana.com Dashboards .

3. Extending Monitoring to Other Services

3.1 MySQL Monitoring

Install MySQL, import a sample SQL schema, download mysqld_exporter, create a .my.cnf with credentials, and run: ./mysqld_exporter -config.my-cnf=".my.cnf" & Add to prometheus.yml:

- job_name: 'mysql'
    static_configs:
      - targets: ['<em>mysql_ip</em>:9104']

3.2 RabbitMQ Monitoring

Download and run rabbitmq_exporter (default port 9099). Add to prometheus.yml:

- job_name: 'RabbitMQ'
  static_configs:
    - targets: ['<em>rabbitmq_ip</em>:9099']

Import dashboard ID 2121 in Grafana.

3.3 Redis Monitoring

Download and run redis_exporter (default port 9121). Add to prometheus.yml:

- job_name: redis
  static_configs:
    - targets: ['<em>redis_ip</em>:9121']

Import dashboard ID 731.

3.4 TiDB Monitoring

Expose TiDB metrics on port 10080 and add to prometheus.yml:

- job_name: 'tidb'
  honor_labels: true
  static_configs:
    - targets: ['<em>tidb_ip</em>:10080']

Import the TiDB JSON dashboard (tidb.json) in Grafana.

4. Grafana Alerting via Email

Configure SMTP settings in grafana.ini (e.g., smtp.qq.com:465 for QQ mail). Create a notification channel, define alert rules on dashboards, and test that emails are received.

5. Grafana HTTP API and API Keys

Grafana’s admin HTTP API requires basic authentication; API keys are scoped to organizations and roles (Viewer, Editor, Admin). Create an API key in the Grafana UI, then use it as a Bearer token in HTTP requests to manage dashboards, data sources, etc.