How to Deploy a Consul Cluster and Enable Prometheus Service Discovery

Consul is a distributed, highly‑available service‑registration system composed of server and client nodes; clients forward requests to servers, and servers achieve data consistency over LAN or WAN.

Prometheus Service Discovery via Consul

The discovery process works as follows:

Register or deregister services (monitoring targets) in Consul.

Prometheus continuously watches Consul; when matching services change, Prometheus updates its scrape targets.

Installing a Consul Cluster

Three virtual machines are created with 4 CPU, 6 GB RAM, 60 GB disk. Hostnames and IPs are:

master1 – 192.168.124.16

master2 – 192.168.124.26

node1 – 192.168.124.56

Deploy Consul on the three nodes (master1 as server, master2 and node1 as clients).

1. Download Consul binary

mkdir /opt/consul/data-p && cd /opt/consul
wget https://releases.hashicorp.com/consul/1.7.1/consul_1.7.1_linux_amd64.zip
unzip consul_1.7.1_linux_amd64.zip

All versions are available at https://releases.hashicorp.com/consul; 1.7.1 is used here.

2. Start Consul agents

On master1 (server):

cd /opt/consul
./consul agent -server -bootstrap -bind=192.168.124.16 -client=192.168.124.16 -data-dir=data -ui -node=192.168.124.16

On master2 (client):

cd /opt/consul
./consul agent -bind=192.168.124.26 -client=192.168.124.26 -data-dir=data -node=192.168.124.26 -join=192.168.124.16

On node1 (client):

cd /opt/consul
./consul agent -bind=192.168.124.56 -client=192.168.124.56 -data-dir=data -node=192.168.124.56 -join=192.168.124.16

After all agents start, open http://192.168.124.16:8500/ in a browser to view the Consul UI.

3. Register a service in Consul

Use the HTTP API to register node-exporter on master1 (assuming node‑exporter is already running on that node):

curl -X PUT -d '{"id":"node-exporter","name":"node-exporter","address":"192.168.124.16","port":9100,"tags":["node-exporter"],"checks":[{"http":"http://192.168.124.16:9100/","interval":"5s"}]}' http://192.168.124.16:8500/v1/agent/service/register

4. Deregister a service

curl --request PUT http://192.168.124.16:8500/v1/agent/service/deregister/192.168.124.16

5. Configure Prometheus to discover services via Consul

On the machine running Prometheus, edit prometheus.yaml (or the appropriate scrape config file) and add the following block:

scrape_configs:
  - job_name: consul
    metrics_path: /metrics
    scheme: http
    consul_sd_configs:
      - server: 192.168.124.16:8500
        services: []
    relabel_configs:
      - source_labels: ['__meta_consul_tags']
        target_label: 'product'
      - source_labels: ['__meta_consul_dc']
        target_label: 'idc'
      - source_labels: ['__meta_consul_service']
        regex: "consul"
        action: drop
      - source_labels: ['job']
        target_label: 'environment'
        regex: '(.*)job'
        replacement: '${1}'

If Prometheus runs in Docker, restart it with:

docker restart prometheus
docker restart node-exporter

6. Verify discovery in the Prometheus UI

Open http://192.168.124.16:9090/targets#job-prometheus. The UI should list the Consul‑discovered targets, confirming that Prometheus is pulling data from Consul.

7. Explanation of relabel fields

__meta_consul_tags

: list of tags attached to the target, separated by delimiters. __meta_consul_dc: name of the Consul data center. __meta_consul_service: name of the service. job: job label of the target server. __meta_consul_service_port: port of the service.

Static configurations define the data source, while consul_sd_configs enable Consul‑based discovery, and relabel_configs transform metadata into Prometheus labels.