Master Blackbox Exporter: Install, Configure, and Monitor with Prometheus

Overview

Monitoring can be divided into white‑box (internal metrics) and black‑box (external symptoms). White‑box focuses on causes such as Redis info or MySQL variables, while black‑box watches user‑visible failures like HTTP errors or service downtime.

Blackbox Exporter

Prometheus provides the official Blackbox Exporter to probe services via HTTP, HTTPS, DNS, TCP and ICMP. It is used in the author’s production environment for network health checks.

Application Scenarios

HTTP test : define request headers, verify status, headers and body.

TCP test : monitor service ports and application‑layer protocols.

ICMP test : host reachability.

POST test : check API connectivity.

SSL certificate expiration : monitor cert expiry dates.

Grafana Templates

Custom Grafana dashboards (e.g., template 9965) visualize SSL monitoring, network line status, and interface health. Sample images illustrate the dashboards.

Blackbox Exporter Deployment

1. Install the exporter:

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
 tar -zxvf blackbox_exporter-0.16.0.linux-amd64.tar.gz -C /usr/local
 mv /usr/local/blackbox_exporter-0.16.0.linux-amd64 /usr/local/blackbox_exporter

2. Add a systemd service:

[Unit]
Description=blackbox_exporter
After=network.target

[Service]
WorkingDirectory=/usr/local/blackbox
ExecStart=/usr/local/blackbox/blackbox_exporter \
  --config.file=/usr/local/blackbox/blackbox.yml

[Install]
WantedBy=multi-user.target

3. Verify the service is listening on port 9115:

ss -tunlp | grep 9115

ICMP Monitoring

ICMP probes help identify faulty network paths. Two approaches are mentioned: paid third‑party services or self‑hosted ping nodes. The author uses self‑hosted ping nodes and configures Prometheus as follows:

- job_name: "icmp_ping"
  metrics_path: /probe
  params:
    module: [icmp]
  file_sd_configs:
    - refresh_interval: 10s
      files: ["/home/prometheus/conf/ping_status*.yml"]
  relabel_configs:
    - source_labels: [__address__]
      regex: (.*)(:80)?
      target_label: __param_target
      replacement: ${1}
    - source_labels: [__param_target]
      target_label: instance
    - source_labels: [__param_target]
      regex: (.*)
      target_label: ping
      replacement: ${1}
    - source_labels: []
      regex: .*
      target_label: __address__
      replacement: 192.168.1.14:9115

Ping target list (ping_status.yml) defines groups of IPs for different ISPs.

- targets: ['220.181.38.150','14.215.177.39','180.101.49.12']
  labels:
    group: '一线城市-电信网络监控'
- targets: ['112.80.248.75','163.177.151.109']
  labels:
    group: '一线城市-联通网络监控'
- targets: ['183.232.231.172','36.152.44.95']
  labels:
    group: '一线城市-移动网络监控'

HTTP Monitoring

Prometheus job for generic HTTP checks:

- job_name: "blackbox"
  metrics_path: /probe
  params:
    module: [http_2xx]
  file_sd_configs:
    - refresh_interval: 1m
      files: ["/home/prometheus/conf/blackbox*.yml"]
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 192.168.1.14:9115

Target list (blackbox-dis.yml) includes example URLs such as https://www.zhibo8.cc and https://www.baidu.com.

- targets:
  - https://www.zhibo8.cc
  - https://www.baidu.com

GET Request Monitoring

Job definition mirrors the generic HTTP job, using a service_get.yml file that lists internal API endpoints.

- job_name: "check_get"
  metrics_path: /probe
  params:
    module: [http_2xx]
  file_sd_configs:
    - refresh_interval: 1m
      files: ["/home/prometheus/conf/service_get.yml"]
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 192.168.1.14:9115

service_get.yml example:

- targets:
  - http://10.10.1.123:10000/pmkb/atc_tcbi
  - http://10.10.1.123:10000/pmkb/get_ship_lock_count
  # … more endpoints …
  labels:
    group: 'service'

POST Request Monitoring

Define a custom module in blackbox.yml for POST requests:

modules:
  http_2xx:
    prober: http
  http_post_2xx:
    prober: http
    http:
      method: POST
      headers:
        Content-Type: application/json
      body: '{"username":"admin","password":"123456"}'

Add a Prometheus job that references the new module:

- job_name: "check_service"
  metrics_path: /probe
  params:
    module: [http_post_2xx]
  file_sd_configs:
    - refresh_interval: 1m
      files: ["/home/prometheus/conf/service_post.yml"]
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 192.168.1.14:9115

service_post.yml example:

- targets:
  - http://10.2.4.103:5000/devops/api/v1.0/login
  labels:
    group: 'service'

TCP Port Monitoring

Prometheus job for TCP connectivity checks:

- job_name: 'port_status'
  metrics_path: /probe
  params:
    module: [tcp_connect]
  static_configs:
    - targets: ['10.10.1.35:8068','10.10.1.35:8069']
  labels:
    instance: 'port_status'
    group: 'tcp'
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - target_label: __address__
      replacement: 192.168.1.14:9115

Alert Rules

Common alert condition: probe_success == 0 indicates connectivity failure. Example rule for generic network issues:

groups:
- name: blackbox_network_stats
  rules:
  - alert: blackbox_network_stats
    expr: probe_success == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Interface/host/port {{ $labels.instance }} unreachable"
      description: "Please investigate immediately."

SSL certificate expiry alert (triggered when less than 30 days remain):

groups:
- name: check_ssl_status
  rules:
  - alert: "ssl证书过期警告"
    expr: (probe_ssl_earliest_cert_expiry - time())/86400 < 30
    for: 1h
    labels:
      severity: warn
    annotations:
      description: 'Domain {{ $labels.instance }} certificate expires in {{ printf "%.1f" $value }} days, please renew.'
      summary: "ssl证书过期警告"

Summary

Black‑box monitoring differs from white‑box by focusing on fault detection from the user’s perspective, allowing rapid identification of issues such as failed ports, interfaces, or network paths. Using Prometheus Blackbox Exporter, operators can quickly set up custom probes, visualise results in Grafana, and define precise alerting rules for reliable service health.