Deploy a StatefulSet Prometheus & Alertmanager Cluster with Persistent Storage on Kubernetes

This article explains how to manually deploy a StatefulSet‑based Prometheus and Alertmanager cluster on Kubernetes, using a StorageClass to provide persistent storage for monitoring data.

It begins by noting that many persistence solutions exist (Thanos, M3DB, InfluxDB, VictoriaMetrics) and chooses a StorageClass‑based approach for this tutorial.

Environment

The deployment is performed on a local test cluster created with

sealos

. The nodes run Ubuntu 18.04 with Kubernetes version 1.17.7. Four master nodes (sealos‑k8s‑m1 to m3) and two worker nodes (sealos‑k8s‑node1 to node2) are used, each exposing services such as node‑exporter and Prometheus.

Label Nodes

<code># Add labels for Prometheus and Alertmanager
kubectl label node sealos-k8s-node1 k8s-app=prometheus
kubectl label node sealos-k8s-node2 k8s-app=prometheus
kubectl label node sealos-k8s-node3 k8s-app=prometheus
kubectl label node sealos-k8s-m1 k8s-app=prometheus-federate
kubectl label node sealos-k8s-m2 k8s-app=alertmanager
kubectl label node sealos-k8s-m3 k8s-app=alertmanager</code>

Create a directory to hold all deployment manifests:

<code>mkdir -p /data/manual-deploy/{prometheus,alertmanager,node-exporter,kube-state-metrics}
</code>

Deploy Prometheus

Create a StorageClass for Prometheus data:

<code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: prometheus-lpv
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
</code>

Define PersistentVolumes for each node, binding them to the StorageClass and specifying node affinity:

<code># Example PV for node1
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-lpv-0
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: prometheus-lpv
  local:
    path: /data/prometheus
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - sealos-k8s-node1
</code>

Provide RBAC rules, a ConfigMap with

prometheus.yml

, and a Service definition. Then create the StatefulSet that runs the Prometheus container with appropriate arguments, volume mounts, probes, and resource limits.

<code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
  namespace: kube-system
  labels:
    k8s-app: prometheus
spec:
  serviceName: "prometheus"
  podManagementPolicy: "Parallel"
  replicas: 3
  selector:
    matchLabels:
      k8s-app: prometheus
  template:
    metadata:
      labels:
        k8s-app: prometheus
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: k8s-app
                operator: In
                values:
                - prometheus
            topologyKey: "kubernetes.io/hostname"
      priorityClassName: system-cluster-critical
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: prometheus
        image: prom/prometheus:v2.20.0
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--storage.tsdb.path=/prometheus"
        - "--storage.tsdb.retention=24h"
        - "--web.console.libraries=/etc/prometheus/console_libraries"
        - "--web.console.templates=/etc/prometheus/consoles"
        - "--web.enable-lifecycle"
        ports:
        - containerPort: 9090
          protocol: TCP
        volumeMounts:
        - name: prometheus-data
          mountPath: "/prometheus"
        - name: config-volume
          mountPath: "/etc/prometheus"
        readinessProbe:
          httpGet:
            path: /-/ready
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
        livenessProbe:
          httpGet:
            path: /-/healthy
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 1000m
            memory: 2500Mi
        securityContext:
          runAsUser: 65534
          privileged: true
      serviceAccountName: prometheus
      volumes:
      - name: config-volume
        configMap:
          name: prometheus-config
  volumeClaimTemplates:
  - metadata:
      name: prometheus-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "prometheus-lpv"
      resources:
        requests:
          storage: 5Gi
</code>

Apply all manifests:

<code>cd /data/manual-deploy/prometheus
kubectl apply -f .
</code>

Verify that PersistentVolumes are bound and that Prometheus pods are running:

<code>kubectl get pv
kubectl -n kube-system get pvc
kubectl -n kube-system get pod -l k8s-app=prometheus
</code>

Deploy Node Exporter

Create a DaemonSet and Service to expose host metrics:

<code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: kube-system
  labels:
    k8s-app: node-exporter
spec:
  selector:
    matchLabels:
      k8s-app: node-exporter
  template:
    metadata:
      labels:
        k8s-app: node-exporter
    spec:
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      containers:
      - name: prometheus-node-exporter
        image: quay.io/prometheus/node-exporter:v1.0.0
        ports:
        - containerPort: 9100
          hostPort: 9100
          protocol: TCP
          name: metrics
        volumeMounts:
        - name: proc
          mountPath: /host/proc
        - name: sys
          mountPath: /host/sys
        - name: rootfs
          mountPath: /host
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: rootfs
        hostPath:
          path: /
      hostNetwork: true
      hostPID: true
---
apiVersion: v1
kind: Service
metadata:
  name: node-exporter
  namespace: kube-system
  labels:
    k8s-app: node-exporter
  annotations:
    prometheus.io/scrape: "true"
spec:
  ports:
  - name: http
    port: 9100
    protocol: TCP
  selector:
    k8s-app: node-exporter
</code>

Apply and verify the DaemonSet pods are running.

<code>kubectl apply -f node-exporter.yaml
kubectl -n kube-system get pod -l k8s-app=node-exporter
</code>

Deploy kube‑state‑metrics

Set up RBAC, a Deployment, and a Service that expose cluster‑level resource metrics for Prometheus scraping.

<code># RBAC (Role, RoleBinding, ClusterRole, ClusterRoleBinding) omitted for brevity
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: quay.io/coreos/kube-state-metrics:v1.6.0
        ports:
        - name: http-metrics
          containerPort: 8080
        - name: telemetry
          containerPort: 8081
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
      - name: addon-resizer
        image: k8s.gcr.io/addon-resizer:1.8.4
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        command:
        - /pod_nanny
        - --container=kube-state-metrics
        - --cpu=100m
        - --extra-cpu=1m
        - --memory=100Mi
        - --extra-memory=2Mi
        - --threshold=5
        - --deployment=kube-state-metrics
---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
  annotations:
    prometheus.io/scrape: "true"
spec:
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
    protocol: TCP
  - name: telemetry
    port: 8081
    targetPort: telemetry
    protocol: TCP
  selector:
    k8s-app: kube-state-metrics
</code>

Apply and verify the pods:

<code>kubectl apply -f kube-state-metrics-rbac.yaml
kubectl apply -f kube-state-metrics-deployment.yaml
kubectl -n kube-system get pod -l k8s-app=kube-state-metrics
</code>

Deploy Alertmanager Cluster

Create a StorageClass and PersistentVolumes for Alertmanager data, then define a ConfigMap with alert routing and email settings.

<code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alertmanager-lpv
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
</code>

<code># Example PV for node m2
apiVersion: v1
kind: PersistentVolume
metadata:
  name: alertmanager-pv-0
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: alertmanager-lpv
  local:
    path: /data/alertmanager
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - sealos-k8s-m2
</code>

<code>apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: EnsureExists
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
      smtp_smarthost: 'smtp.qq.com:465'
      smtp_from: '[email protected]'
      smtp_auth_username: '[email protected]'
      smtp_auth_password: 'bhgb'
      smtp_hello: '警报邮件'
      smtp_require_tls: false
    route:
      group_by: ['alertname','cluster']
      group_wait: 30s
      group_interval: 30s
      repeat_interval: 12h
      receiver: default
      routes:
      - receiver: email
        group_wait: 10s
        match:
          team: ops
    receivers:
    - name: default
      email_configs:
      - to: '[email protected]'
        send_resolved: true
    - name: email
      email_configs:
      - to: '[email protected]'
        send_resolved: true
</code>

Define a StatefulSet (cluster mode) with two replicas, peer discovery, and volume claims, plus an accompanying headless Service for mesh communication.

<code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: alertmanager
  namespace: kube-system
  labels:
    k8s-app: alertmanager
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    version: v0.21.0
spec:
  serviceName: "alertmanager-operated"
  replicas: 2
  selector:
    matchLabels:
      k8s-app: alertmanager
      version: v0.21.0
  template:
    metadata:
      labels:
        k8s-app: alertmanager
        version: v0.21.0
    spec:
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: k8s-app
                operator: In
                values:
                - alertmanager
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: prometheus-alertmanager
        image: "prom/alertmanager:v0.21.0"
        args:
        - "--config.file=/etc/config/alertmanager.yml"
        - "--storage.path=/data"
        - "--cluster.listen-address=${POD_IP}:9094"
        - "--web.listen-address=:9093"
        - "--cluster.peer=alertmanager-0.alertmanager-operated:9094"
        - "--cluster.peer=alertmanager-1.alertmanager-operated:9094"
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 9093
          name: web
          protocol: TCP
        - containerPort: 9094
          name: mesh-tcp
          protocol: TCP
        - containerPort: 9094
          name: mesh-udp
          protocol: UDP
        readinessProbe:
          httpGet:
            path: /#/status
            port: 9093
          initialDelaySeconds: 30
          timeoutSeconds: 60
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
        - name: storage-volume
          mountPath: "/data"
        resources:
          limits:
            cpu: 1000m
            memory: 500Mi
          requests:
            cpu: 10m
            memory: 50Mi
        securityContext:
          runAsUser: 0
          privileged: true
      - name: prometheus-alertmanager-configmap-reload
        image: "jimmidyson/configmap-reload:v0.4.0"
        args:
        - --volume-dir=/etc/config
        - --webhook-url=http://localhost:9093/-/reload
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
          readOnly: true
        resources:
          limits:
            cpu: 10m
            memory: 10Mi
          requests:
            cpu: 10m
            memory: 10Mi
        securityContext:
          runAsUser: 0
          privileged: true
      volumes:
      - name: config-volume
        configMap:
          name: alertmanager-config
  volumeClaimTemplates:
  - metadata:
      name: storage-volume
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "alertmanager-lpv"
      resources:
        requests:
          storage: 5Gi
</code>

<code>apiVersion: v1
kind: Service
metadata:
  name: alertmanager-operated
  namespace: kube-system
  labels:
    app.kubernetes.io/name: alertmanager-operated
    app.kubernetes.io/component: alertmanager
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    k8s-app: alertmanager
  ports:
  - name: web
    port: 9093
    targetPort: web
    protocol: TCP
  - name: tcp-mesh
    port: 9094
    targetPort: tcp-mesh
    protocol: TCP
  - name: udp-mesh
    port: 9094
    targetPort: udp-mesh
    protocol: UDP
</code>

Apply all Alertmanager manifests and confirm the pods are running.

<code>cd /data/manual-deploy/alertmanager
kubectl apply -f .
kubectl -n kube-system get pod -l k8s-app=alertmanager
</code>

At this point a fully functional, highly available Prometheus and Alertmanager stack is running in the

kube-system

namespace, ready to be scraped by Grafana or other visualization tools.