Deploy and Manage Prometheus Operator on Kubernetes: A Step‑by‑Step Guide

What is Prometheus Operator?

Prometheus is a stateful monitoring system that stores its configuration in ConfigMaps. Updating the configuration traditionally required manually deleting Pods so that Kubernetes could recreate them with the new ConfigMap, which is cumbersome for large deployments. The Operator pattern, introduced by CoreOS, automates the lifecycle of complex stateful applications by extending the Kubernetes API with custom resources and controllers. Prometheus Operator implements this pattern for Prometheus.

How Prometheus Operator Works

The Operator adds custom resource definitions (CRDs) and a controller that watches those resources. When a user creates or updates a CRD, the controller reconciles the desired state by creating or updating native Kubernetes objects such as Deployment, StatefulSet, Service, ConfigMap, and Secret. This declarative approach lets users focus on the final state of the monitoring stack rather than the low‑level steps required to achieve it.

CRDs Provided by Prometheus Operator

Prometheus : declaratively creates and manages a Prometheus server instance.

ServiceMonitor : declaratively defines how to scrape metrics from a set of Services.

PrometheusRule : declaratively manages alerting and recording rules.

Alertmanager : declaratively creates and manages an Alertmanager instance.

Together these resources automate the provisioning and configuration of a full monitoring stack.

Deploying Prometheus Operator

Clone the operator source code:

git clone https://github.com/coreos/prometheus-operator.git

Create a dedicated namespace for monitoring: kubectl create namespace monitoring Apply the bundled manifests (adjust the namespace from default to monitoring if needed):

cd prometheus-operator/</code>
<code>kubectl -n monitoring apply -f bundle.yaml

Verify the operator pods are running: kubectl -n monitoring get pods If the operator image cannot be pulled, load a pre‑downloaded .tar.gz image manually:

docker load -i prometheus-operator.tar.gz

Creating a Prometheus Instance

Save the following manifest as prometheus-inst.yaml and apply it:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: inst
  namespace: monitoring
spec:
  resources:
    requests:
      memory: 400Mi

kubectl apply -f prometheus-inst.yaml

Check that a StatefulSet for the Prometheus server has been created:

kubectl -n monitoring get statefulset

Exposing Prometheus via Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus-operator
  name: prometheus-operator-svc
  namespace: monitoring
spec:
  ports:
  - name: operator
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app: prometheus
    prometheus: inst
  sessionAffinity: None
  type: NodePort

kubectl apply -f prometheus-service.yaml

After the service is created, the UI is reachable at http://<master‑IP>:<NodePort>/config (e.g., http://192.168.0.6:31535/config).

Managing Monitoring Configuration with ServiceMonitor

Deploy a sample application:

kind: Service
apiVersion: v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
  - name: web
    port: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  selector:
    matchLabels:
      app: example-app
  replicas: 3
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: fabxc/instrumented_app
        ports:
        - name: web
          containerPort: 8080

kubectl apply -f example-app.yaml

Create a ServiceMonitor that selects the above Service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  namespace: monitoring
  labels:
    team: frontend
spec:
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: web

kubectl create -f example-app-service-monitor.yaml

To monitor resources across namespaces, set namespaceSelector.any: true. For services protected by BasicAuth, add a basicAuth block referencing a Secret that stores the credentials.

Linking Prometheus to ServiceMonitor

Update the Prometheus CR to select the ServiceMonitor by label:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: inst
  namespace: monitoring
spec:
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi

kubectl -n monitoring apply -f prometheus-inst.yaml

After the change, the Prometheus configuration automatically includes a scrape job for the example app.

Custom ServiceAccount and RBAC

The default ServiceAccount in the monitoring namespace lacks permission to list resources in the default namespace. Create a dedicated ServiceAccount and grant it the necessary cluster‑wide permissions:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources: ["nodes","services","endpoints","pods"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

kubectl -n monitoring apply -f prometheus-rbac.yaml

Reference the ServiceAccount in the Prometheus CR:

spec:
  serviceAccountName: prometheus

Result

After applying the RBAC changes and re‑creating the Prometheus instance, the operator can successfully discover endpoints, and the Prometheus UI shows collected metrics from the example application.