Comprehensive Guide to Installing and Using Prometheus with Grafana for Monitoring

Learning Objectives

Install Prometheus server

Monitor remote Linux with node_exporter

Monitor remote MySQL with mysqld_exporter

Install Grafana and add Prometheus as a data source

Create Grafana dashboards for CPU load and MySQL metrics

Implement alerting using Grafana + onealert

Task Background

A fast‑growing e‑commerce company requires continuous business monitoring, assigning the operations team to implement the project.

Task Requirements

Deploy a 24/7 monitoring server

Design monitoring items and alert rules for business and R&D

Establish a timely warning mechanism

Implement multi‑level alerts: phone, WeChat, email

Handle centralized monitoring for remote servers, especially Kubernetes clusters using Prometheus

Why Monitor? Real‑time data collection and alerts help detect and resolve issues promptly, providing a basis for performance optimization.

Four Elements of Monitoring

Monitoring objects (hosts, services, resources, URLs)

Monitoring tool – Prometheus

Monitoring schedule (7×24, 5×8)

Alert recipients (administrators)

Prometheus Overview

Prometheus, written in Go, is an open‑source monitoring, alerting, and time‑series database solution, especially suited for Docker containers and Kubernetes environments. https://prometheus.io/docs/introduction/overview/ Time‑Series Data

Time‑series data records system or device state changes over time. It is widely used in autonomous vehicles, fleet tracking, real‑time trading, and operational monitoring.

Characteristics of Time‑Series Databases

High performance for large‑scale data

Low storage cost due to efficient compression (≈3.5 bytes per sample)

Key Features of Prometheus

Multi‑dimensional data model

Flexible query language (PromQL)

Self‑contained server node (no external storage dependency)

Pull‑based data collection via HTTP

Optional push via gateway

Service discovery or static configuration

Rich visualization options

Prometheus Architecture

See the diagram below (image omitted for brevity).

Experimental Environment Preparation

Static IP with internet access

Hostname configuration

Time synchronization

Disable firewall and SELinux

1️⃣ Install Prometheus

Download the binary from the official site and extract it. https://prometheus.io/download/ Run the server and access the UI at http://<em>SERVER_IP</em>:9090. By default only the local host is monitored.

2️⃣ Monitor Remote Linux Host

On the target host (agent1) install node_exporter and start it (use nohup to keep it running after logout).

Access http://<em>AGENT_IP</em>:9100/metrics to verify data collection.

Add the target to prometheus.yml on the server; the UI will show the new target under Status → Targets.

3️⃣ Monitor Remote MySQL

Install mysqld_exporter on the MySQL host, then add its scrape configuration to prometheus.yml. Verify the target appears in the Prometheus UI.

Grafana Visualization

Grafana is an open‑source analytics and visualization tool that can query Prometheus data and generate dashboards with alerting capabilities. https://grafana.com/ Install Grafana, access http://<em>GRAFANA_IP</em>:3000 (default admin/admin), and add Prometheus as a data source.

Create dashboards to display CPU load, MySQL metrics, etc., and save them.

Grafana + onealert Alerting

Instead of Prometheus Alertmanager, use Grafana integrated with onealert for easier alert configuration. Add Grafana as an application in onealert, set up notification policies, and configure Grafana notification channels.

Test alerts by creating a rule for CPU load; adjust thresholds if needed.

Common Alerting Pitfalls

Time synchronization across servers

Alert notification content must not be empty

Save changes after editing rules

Alert state must transition from OK to alerting

Ensure Grafana can communicate with onealert

Conclusion

This guide demonstrates end‑to‑end monitoring setup using Prometheus, node_exporter, mysqld_exporter, and Grafana, along with practical alerting via onealert, providing a solid foundation for reliable operations monitoring.