How to Monitor Hybrid Cloud with Prometheus: Fix Missing Physical Machine Targets

Problem Scenario Reconstruction

A user wants a single Prometheus job to monitor both Kubernetes‑deployed node-exporter (exposed via endpoints) and a directly deployed node-exporter on a physical machine (IP:Port).

Initial Configuration

- job_name: node-exporter
  static_configs:
  - targets: ["172.139.20.1:9100"] # physical machine address
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_endpoints_name,__meta_kubernetes_endpoint_port_name]
    regex: kube-system;node-exporter;metrics

Result: the physical‑machine target 172.139.20.1:9100 could not be scraped.

Root Cause Analysis

Prometheus merges all discovery mechanisms (static_configs + kubernetes_sd_configs) under the same job.

The keep action in relabel_configs requires targets to match the Kubernetes metadata label kube-system;node-exporter;metrics.

Physical‑machine targets lack any __meta_kubernetes_* labels, so they are filtered out.

Solution: Separate Handling + Dynamic Labeling

Core idea: use relabel rules to distinguish Kubernetes and physical‑machine targets and treat them separately.

Optimized Configuration

- job_name: node-exporter
  static_configs:
  - targets: ["172.139.20.1:9100"]
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - source_labels: [__address__]
    action: replace
    regex: (.*):10250
    target_label: __address__
    replacement: $1:9100
  - source_labels: [__address__]
    target_label: instance

Key Configuration Interpretation

Physical‑machine static config: define targets directly with static_configs to avoid being filtered by Kubernetes relabel rules.

Kubernetes dynamic discovery: use the node role to automatically discover cluster nodes.

The replace action changes the scraped metrics port from the default 10250 to 9100.

Both discovery mechanisms apply a second relabel rule that copies the __address__ value to the instance label.

Effect Verification

Visit the Prometheus targets page (e.g., http://<prometheus>:9090/targets) to confirm that both the Kubernetes and physical‑machine node-exporter targets appear and are being scraped.

Conclusion

By applying appropriate relabel rules and label strategies, Prometheus can seamlessly monitor hybrid environments. The provided configuration has been validated in production and can be reused or adapted to fit specific labeling requirements.