Prometheus Service Discovery: File‑Based, API, and DNS Methods

Manual configuration of monitoring targets is impractical for large‑scale environments, especially Kubernetes. Prometheus solves this with service discovery, an automatic mechanism that detects, classifies, and identifies new or changed targets. It supports three discovery methods: files generated by configuration‑management tools, API queries (e.g., Amazon AWS API), and DNS records.

File‑based discovery is a modest step beyond static configuration and is useful when configuration‑management systems (such as Puppet, Ansible, or a CMDB) generate target lists in YAML or JSON files. Prometheus periodically reloads these files according to a defined schedule, allowing dynamic updates without restarting the service.

Below is an example of a Prometheus configuration file ( /etc/prometheus/prometheus.yml ) that defines global settings, alerting, and scrape configurations using file‑based service discovery:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            # - alertmanager:9093

scrape_configs:
  - job_name: 'node'
    file_sd_configs:
      - files:
          - targets/nodes/*.json
        refresh_interval: 5m

  - job_name: 'docker'
    file_sd_configs:
      - files:
          - targets/docker/*.yml
        refresh_interval: 5m

After editing the configuration, you can verify its correctness with the Prometheus tool:

# promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
SUCCESS: 0 rule files found

These steps illustrate how Prometheus can automatically discover and monitor targets using flexible, file‑driven configurations suitable for dynamic, container‑orchestrated environments.