Master Prometheus: From Basics to Advanced Configuration and Alerts

1. Introduction

Prometheus is an open‑source monitoring and alerting system originally developed at Google. It stores time‑series metric data, provides the PromQL query language, and integrates with many visualization tools such as Grafana.

2. Core components

Key components include:

Prometheus Server : collects and stores metrics, serves PromQL queries.

Client SDKs : libraries for many languages to expose metrics.

Pushgateway : buffers short‑lived metrics for pull‑based collection.

PromDash : a Rails‑based dashboard for visualizing metrics.

Exporter : agents that expose target metrics in Prometheus format.

Alertmanager : handles alert routing, silencing and notification (email, DingTalk, webhook, etc.).

prometheus_cli : command‑line utility.

Other helper tools for converting data to formats used by HAProxy, StatsD, Graphite, etc.

These components work together to collect, process and display monitoring data.

3. Installation

Download and extract the binary:

cd /opt
wget -c https://github.com/prometheus/prometheus/releases/download/v2.14.0/prometheus-2.14.0.linux-amd64.tar.gz
tar zxvf prometheus-2.14.0.linux-amd64.tar.gz

3.1 Configure server

Edit prometheus.yml to define scrape jobs and rule files. Example snippet:

alerting:
  alertmanagers:
  - static_configs:
    - targets: ["10.81.28.227:9093"]
rule_files:
  - "sys_monit.yml"
  - "kafka_monit.yml"
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'system'
    file_sd_configs:
    - refresh_interval: 1m
      files:
       - sys.yml
  - job_name: 'kafka'
    file_sd_configs:
    - refresh_interval: 1m
      files:
       - kafka.yml

Example sys.yml defines target hosts:

[
  {
    "labels": {
      "job": "shop001.web.pub.pro.ali.dc",
      "instance": "system"
    },
    "targets": ["10.174.88.9:9100"]
  }
]

Start Prometheus:

nohup ./prometheus &

3.2 Alertmanager configuration

Download Alertmanager from its GitHub releases and configure alertmanager.yml. The routing section may look like:

global:
  resolve_timeout: 5m
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 10m
  receiver: 'pro'
  routes:
  - receiver: 'canal'
    match:
      alertname: 'canal 延迟'
receivers:
- name: 'sys'
  webhook_configs:
  - url: 'http://10.81.28.227:8060/dingtalk/pro/send'
- name: 'canal'
  webhook_configs:
  - url: 'http://10.81.28.227:8060/dingtalk/canal/send'

Alert rules are stored in separate files, e.g. sys_monit.yml:

groups:
  - name: netstat_tcp_time_wait
    rules:
    - alert: 主机连接数 time_wait
      expr: netstat_tcp_time_wait > 60000
      for: 5m
      labels:
        status: warning
      annotations:
        summary: "{{$labels.job}}"
        description: "{{ $labels.instance }} time_wait > 60k (current: {{ $value }})"

4. Integration with other tools

Telegraf can act as a Prometheus client:

[[outputs.prometheus_client]]
    listen = "0.0.0.0:9100"

Grafana is commonly used for dashboards; screenshots illustrate typical views.

Prometheus also integrates with Spring Boot, Canal, and many exporters.

5. Summary

Prometheus offers a powerful, pull‑based monitoring stack with a built‑in time‑series database, flexible query language, and extensive ecosystem. Configuration can become complex for large fleets, so automation scripts and version‑controlled configuration repositories are recommended. The full set of configuration files is available at https://github.com/xjjdog/prometheus-cnf-pro .