Managing Prometheus Alerts and Alertmanager with the Prometheus Operator

Defining Alerting Rules with PrometheusRule

In the Prometheus Operator, alert rules are expressed as a PrometheusRule custom resource instead of a static file. Example manifest:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: example
    role: alert-rules
  name: prometheus-example-rules
spec:
  groups:
  - name: ./example.rules
    rules:
    - alert: ExampleAlert
      expr: vector(1)

Apply the rule:

kubectl -n monitoring apply -f example-rule.yaml

Reference the rule from a Prometheus instance via ruleSelector:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: inst
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  ruleSelector:
    matchLabels:
      role: alert-rules
      prometheus: example
  resources:
    requests:
      memory: 400Mi

After applying, the rule appears in the Prometheus UI.

Managing Alertmanager Instances

Create an Alertmanager custom resource. The example below deploys three replicas:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: inst
  namespace: monitoring
spec:
  replicas: 3

Apply it:

kubectl -n monitoring apply -f alertmanager-inst.yaml

If the pods stay in ContainerCreating, the operator expects a secret named alertmanager-inst that contains the Alertmanager configuration. Create alertmanager.yaml (example configuration):

global:
  resolve_timeout: 5m
route:
  group_by: ['job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'webhook'
receivers:
- name: 'webhook'
  webhook_configs:
  - url: 'http://alertmanagerwh:30500/'

Store the file in a secret:

kubectl -n monitoring create secret generic alertmanager-inst --from-file=alertmanager.yaml

After the secret exists, the Alertmanager pods reach Running. Expose the service (NodePort) to access the UI:

apiVersion: v1
kind: Service
metadata:
  name: alertmanager-operator-svc
  namespace: monitoring
  labels:
    app: alertmanager-service
spec:
  ports:
  - name: operator
    port: 9093
    protocol: TCP
    targetPort: 9093
  selector:
    alertmanager: inst
    app: alertmanager
  type: NodePort

Retrieve the NodePort:

kubectl get svc -n monitoring | grep alertmanager-operator-svc

Access the UI at http://<master-node-ip>:<nodePort>/#/status.

Linking Alertmanager to Prometheus

Update the Prometheus spec to reference the Alertmanager instance:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: inst
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  ruleSelector:
    matchLabels:
      role: alert-rules
      prometheus: example
  alerting:
    alertmanagers:
    - name: alertmanager-example
      namespace: monitoring
      port: web
  resources:
    requests:
      memory: 400Mi

Apply the change; Prometheus reloads and discovers the Alertmanager via service discovery.

Using Custom Prometheus Configuration

When a custom prometheus.yaml is required, create a minimal Prometheus object without any monitoring‑related fields:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: inst-cc
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  resources:
    requests:
      memory: 400Mi

Create the instance:

kubectl -n monitoring create -f prometheus-inst-cc.yaml

The operator creates a secret named prometheus-inst-cc. Edit the secret to add a base64‑encoded custom configuration:

# Encode your prometheus.yaml
cat prometheus.yaml | base64
# Example output (base64 string)
Z2xvYmFsOgogIHNjcmFwZV9pbnRlcnZhbDogMTBzCiAgc2NyYXBlX3RpbWVvdXQ6IDEwcwogIGV2YWx1YXRpb25faW50ZXJ2YWw6IDEwcw==

# Edit the secret
kubectl -n monitoring edit secret prometheus-inst-cc
# Add under data:
prometheus.yaml: "Z2xvYmFsOgogIHNjcmFwZV9pbnRlcnZhbDogMTBzCiAgc2NyYXBlX3RpbWVvdXQ6IDEwcwogIGV2YWx1YXRpb25faW50ZXJ2YWw6IDEwcw=="

Port‑forward to verify the custom configuration is loaded:

kubectl -n monitoring port-forward statefulsets/prometheus-inst-cc 9091:9090

These steps illustrate how the Prometheus Operator enables fully declarative management of Prometheus, alert rules, Alertmanager, and custom configuration files within a Kubernetes cluster.