Mastering Prometheus Service Discovery: File, DNS, and Consul Integration

Types of Prometheus Service Discovery

File‑based discovery

DNS‑based discovery

API‑based discovery (Kubernetes, Consul, Azure…)

Why Prometheus Needs Automatic Discovery

Prometheus pulls metrics, so it must know target locations. Static configs work for small setups, but dynamic cloud environments require automatic discovery to keep target lists up‑to‑date.

Prometheus Scrape Lifecycle

Discovery → Configuration → Relabel → Scrape → Metrics relabel. During each scrape_interval Prometheus generates a target list, attaches __meta_ labels, sets __scheme__, __address__, __metrics_path__, and applies relabeling and metric_relabel_configs.

File‑Based Service Discovery Demo

Define static targets in a YAML file and reload Prometheus.

vi prometheus.yml
# static config nodes
- job_name: 'nodes'
  file_sd_configs:
    - files:
      - targets/nodes-*.yaml
      refresh_interval: 2m
  scrape_interval: 15s

cat targets/nodes-linux.yaml
- targets:
  - monitor.example.com:9100
  - node.export1.com:9101
  - node.export2.com:9101
  - node.export3.com:9101
  labels:
    app: node-exporter
    os: aliyunos3

curl -XPOST monitor.example.com:9090/-/reload

Consul‑Based Service Discovery

Consul provides service registration, health checks, KV store, and multi‑datacenter support.

Deploy Consul with Docker‑Compose

vi docker-compose.yml
version: '3.6'
volumes:
  consul_data: {}
networks:
  monitoring:
    driver: bridge
services:
  consul:
    image: consul:1.14
    volumes:
      - ./consul_configs:/consul/config
      - consul_data:/consul/data/
    networks:
      - monitoring
    ports:
      - 8500:8500
    command: ["consul","agent","-dev","-bootstrap","-config-dir","/consul/config","-data-dir","/consul/data","-ui","-log-level","INFO","-bind","127.0.0.1","-client","0.0.0.0"]
  consul-exporter:
    image: prom/consul-exporter:v0.8.0
    networks:
      - monitoring
    ports:
      - 9107:9107
    command: ["--consul.server=consul:8500"]
    depends_on:
      - consul

Start with docker-compose up -d and verify the containers.

Register Services in Consul

Register via Consul's HTTP API using a JSON definition.

vi grafana.json
{
  "ID":"grafana",
  "Name":"grafana",
  "Tags":["grafana","v9"],
  "Address":"monitor.example.com",
  "Port":3000,
  "Meta":{"grafana_version":"9"},
  "EnableTagOverride":false,
  "Check":{
    "http":"http://monitor.example.com:3000/metrics",
    "interval":"5s",
    "Timeout":"5s"
  },
  "Weights":{"Passing":1,"Warning":1}
}

curl -XPUT --data @grafana.json http://monitor.example.com:8500/v1/agent/service/register

Consul adds __meta_consul_* labels that can be relabeled in Prometheus.

# Example Prometheus job using Consul
- job_name: 'nodes'
  consul_sd_configs:
    - server: "monitor.example.com:8500"
      tags: ["nodes"]
      refresh_interval: 2m
  scrape_interval: 15s

Reload Prometheus after changes:

curl -XPOST monitor.example.com:9090/-/reload

Consul Command‑Line Registration Demo

Prepare a nodes.json file with multiple services, place it in Consul’s data directory, and run consul reload to apply.

{
  "services": [
    {
      "id":"node.export1.com",
      "name":"node.export1.com",
      "address":"node.export1.com",
      "port":9101,
      "tags":["nodes"],
      "checks":[{"http":"http://node.export1.com:9101/metrics","interval":"5s"}]
    },
    {
      "id":"node.export2.com",
      "name":"node.export2.com",
      "address":"node.export2.com",
      "port":9101,
      "tags":["nodes"],
      "checks":[{"http":"http://node.export2.com:9101/metrics","interval":"5s"}]
    }
    // ... other services ...
  ]
}

After reloading, both Consul and Prometheus display the discovered targets.

Thus the basic demonstration of Prometheus automatic discovery via Consul is complete.