Master Prometheus-Operator: Install, Configure ServiceMonitor, and Monitor Apps on Kubernetes

Background

In non‑operator setups, monitoring a Kubernetes cluster is usually done via ConfigMap‑based service discovery and metric scraping. Switching to Prometheus‑Operator introduces new concepts such as ServiceMonitor and PodMonitor, which many users need to learn.

Prometheus‑Operator Introduction

Prometheus‑Operator provides a native way to deploy and manage Prometheus‑related monitoring components on Kubernetes, simplifying and automating the configuration of a Prometheus‑based monitoring stack.

Custom resources: uses Kubernetes CRDs to manage Prometheus, Alertmanager, and related components.

Simplified deployment: configure Prometheus via Kubernetes manifests (version, persistence, replicas, retention, etc.).

Automatic target discovery: generates scrape configurations from Kubernetes labels without writing raw Prometheus config.

2.1 Architecture

The official architecture diagram shows how the Operator acts as the core controller, creating CRD objects such as Prometheus, ServiceMonitor, Alertmanager, and PrometheusRule, and continuously reconciling their desired state.

Three YAML files illustrate how Prometheus links to ServiceMonitor, and how ServiceMonitor selects target Services.

ServiceMonitor decouples the monitoring requirement from its implementation by using label selectors to declare which Endpoints should be scraped, letting users focus on the "what" rather than the "how".

The following diagram shows the resources involved in configuring alerts and their roles.

Typical workflow: configure ServiceMonitor/PodMonitor to collect metrics, set Prometheus.spec.alerting to reference Alertmanager, define PrometheusRule for alerting rules, and configure AlertmanagerConfig for routing, inhibition, and notification.

2.2 Common CRDs

Prometheus : defines the desired Prometheus deployment.

ServiceMonitor : declaratively specifies how to monitor a group of Kubernetes Services; the Operator generates the corresponding scrape config.

PodMonitor : similar to ServiceMonitor but targets Pods directly.

PrometheusRule : defines alerting and recording rules for Prometheus.

Alertmanager : defines the desired Alertmanager deployment.

AlertmanagerConfig : declaratively configures parts of Alertmanager, such as routing to custom receivers and inhibition rules.

Probe : declares how to monitor endpoints or static targets, often used with the Blackbox exporter.

ThanosRuler : defines the desired Thanos Ruler deployment.

Prometheus‑Operator Installation

The Operator requires a specific Kubernetes version; select the matching version from the repository https://github.com/prometheus-operator/kube-prometheus .

This document uses a 1.25 Kubernetes cluster with Prometheus‑Operator version 0.12.0 ( release‑0.12.zip ).

3.1 Installation

wget https://github.com/prometheus-operator/kube-prometheus/archive/refs/heads/release-0.12.zip
unzip release-0.12.zip
cd kube-prometheus-release-0.12
kubectl apply --server-side -f manifests/setup
kubectl wait --for condition=Established --all CustomResourceDefinition --namespace=monitoring
kubectl apply -f manifests/

Note: Images for kube‑state‑metrics and prometheus‑adapter are hosted on Google Container Registry and may be unavailable in some regions; replace them with reachable mirrors if pods remain pending.

3.2 Uninstall

Run the following command only if you want to completely remove Prometheus‑Operator:

kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup

4. Using ServiceMonitor to Monitor Application Metrics

Create a Deployment and Service that expose metrics on port 8080.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sample-metrics-app
  name: sample-metrics-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-metrics-app
  template:
    metadata:
      labels:
        app: sample-metrics-app
    spec:
      containers:
      - image: luxas/autoscale-demo:v0.1.2
        name: sample-metrics-app
        ports:
        - name: web
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: sample-metrics-app
  labels:
    app: sample-metrics-app
spec:
  ports:
  - name: web
    port: 80
    targetPort: 8080
  selector:
    app: sample-metrics-app

Create a ServiceMonitor to collect the application's metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: sample-metrics-app
  labels:
    service-monitor: sample-metrics-app
spec:
  selector:
    matchLabels:
      app: sample-metrics-app  # matches the Service label
  endpoints:
  - port: web  # Prometheus scrapes the port named "web" in the Service

Verify the Service and query the metrics endpoint:

kubectl get service

Access the metrics via curl 10.247.227.116/metrics and observe the http_requests_total metric with a count of 805.

Open the Prometheus UI (using the Prometheus server IP and port) to view the collected metrics and the ServiceMonitor configuration.

Because of the scrape interval, Prometheus shows a count of 800 while the application endpoint reports 805.

Original article: https://www.cnblogs.com/arthinking/p/14450337.html

(Copyright belongs to the original author, removed upon request)