Master ServiceMonitor: Build Reliable Prometheus Monitoring for Kubernetes

In the Kubernetes ecosystem, services are numerous and dynamic, making static monitoring insufficient. The key to letting Prometheus see your services is the ServiceMonitor custom resource.

ServiceMonitor vs Traditional Configuration

Prometheus monitors Kubernetes clusters using kubernetes_sd_config and ServiceMonitor. The former discovers resources via various roles, while ServiceMonitor, a CRD of the Prometheus Operator, encapsulates complex logic into a declarative resource, simplifying configuration and improving readability for large clusters.

Core Fields of ServiceMonitor

Understanding the main fields is essential for accurate and efficient metric collection:

namespaceSelector : Determines which namespaces the selector applies to; can monitor all namespaces (any: true) or a specific list.

selector : Defines which services are monitored via label selectors.

endpoints : Specifies how to scrape metrics from selected services, including port, path, and interval.

relabeling : Allows relabeling or filtering of metric labels before ingestion, helping clean and customize monitoring data.

Hands‑on Examples

Example 1 – Collecting Harbor metrics:

<code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: harbor
  namespace: harbor
  labels:
    release: monitor
spec:
  jobLabel: app.kubernetes.io/name
  namespaceSelector:
    matchNames:
    - harbor
  selector:
    matchLabels:
      release: harbor
      app: harbor
  endpoints:
  - path: /metrics
    port: http-metrics
</code>

Tip: Use

kubectl -n obs-system get prometheus monitor -o jsonpath='{.spec.serviceMonitorSelector}'

to set ServiceMonitor labels.

Example 2 – Collecting GitLab metrics:

<code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gitlab
  namespace: gitlab
  labels:
    release: monitor
spec:
  namespaceSelector:
    matchNames:
    - gitlab
  selector:
    matchLabels:
      release: gitlab
  endpoints:
  - path: /metrics
    port: http-metrics-ws
  - path: /metrics
    port: http-metrics-wh
  - path: /metrics
    port: http-metrics
</code>

Conclusion

ServiceMonitor is a cornerstone of cloud‑native observability. Properly crafted ServiceMonitor resources enable faster issue detection, early warnings, and comprehensive control over your microservice ecosystem.