Step‑by‑Step Guide to Installing, Configuring, and Using Prometheus on CentOS

Download and Run Prometheus

# wget https://github.com/prometheus/prometheus/releases/download/v2.26.0/prometheus-2.26.0.linux-amd64.tar.gz
# tar xvzf prometheus-2.26.0.linux-amd64.tar.gz
# cd prometheus-2.26.0.linux-amd64
# ls
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool

Before starting, edit the configuration file.

Configure Prometheus Self‑Monitoring

Prometheus scrapes its own HTTP endpoint to expose health metrics. Save the following basic configuration as prometheus.yml (the file already exists in the extracted directory).

global:
  scrape_interval: 15s  # default, scrape every 15 seconds
  external_labels:
    monitor: 'codelab-monitor'

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Refer to the official configuration documentation for all options.

Start Prometheus

# ./prometheus --config.file=prometheus.yml

Access the status page at http://localhost:9000 and the metrics endpoint at http://localhost:9090/metrics.

Open Firewall Port

# firewall-cmd --permanent --zone=public --add-port=9090/tcp
success
# firewall-cmd --reload
success

Use the Expression Browser

Open http://localhost:9090/graph, switch to the Table tab (Classic UI > Console) to explore exported metrics such as prometheus_target_interval_length_seconds. Query the 99th‑percentile latency with:

prometheus_target_interval_length_seconds{quantile="0.99"}

Count the number of returned series with:

count(prometheus_target_interval_length_seconds)

Launch Some Scrape Targets

Download and run Node Exporter to provide additional metrics:

# wget https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz
# tar -xvzf node_exporter-1.1.2.linux-amd64.tar.gz
# ./node_exporter --web.listen-address 127.0.0.1:8001
# ./node_exporter --web.listen-address 127.0.0.1:8002
# ./node_exporter --web.listen-address 127.0.0.1:8003

These expose metrics at http://localhost:8001/metrics, http://localhost:8002/metrics and http://localhost:8003/metrics.

Configure Prometheus to Monitor the Example Targets

Add a new job named node that groups the three endpoints, labeling the first two as production and the third as canary:

scrape_configs:
  - job_name: 'node'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:8001', 'localhost:8002']
        labels:
          group: 'production'
      - targets: ['localhost:8003']
        labels:
          group: 'canary'

After updating prometheus.yml, restart Prometheus and verify the new targets under the Targets page.

Create Recording Rules to Aggregate Data

To avoid expensive queries over thousands of series, define a recording rule that calculates the average CPU time per instance over a 5‑minute window:

groups:
  - name: cpu-node
    rules:
      - record: job_instance_mode:node_cpu_seconds:avg_rate5m
        expr: avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))

Save this as prometheus.rules.yml and reference it in prometheus.yml:

rule_files:
  - 'prometheus.rules.yml'

Restart Prometheus and query the new metric job_instance_mode:node_cpu_seconds:avg_rate5m in the expression browser.

Visualize Metrics with the Graph Interface

Use the “Graph” tab to plot expressions such as the rate of chunk creation:

rate(prometheus_tsdb_head_chunks_created_total[1m])

After adding the recording rule, the query returns a persistent series as shown below: