Deploy a Complete Prometheus Monitoring Stack on Kubernetes (Step‑by‑Step)

Introduction

Prometheus is an open‑source monitoring system originally built at SoundCloud and now a CNCF project, making it a natural fit for Kubernetes monitoring.

Prometheus Architecture

Prometheus consists of a server, client libraries, a push gateway, exporters, Alertmanager, and a simple web UI.

All data are stored as time‑series identified by a metric name and a set of key‑value labels.

Jobs and Instances

Instances are scrape targets; a job groups instances that share the same scrape configuration.

Kubernetes Monitoring Metrics

Important metrics include node resource utilization, pod counts, and resource‑object status. Implementations use cAdvisor for pod metrics, node‑exporter for node metrics, and kube‑state‑metrics for object status.

Deploy Prometheus on the K8s Cluster

Cluster environment

Four nodes are used: one master (192.168.73.138) and three workers (192.168.73.136, .139, .140).

Clone the project

git clone https://github.com/zhangdongdong7/k8s-prometheus.git
cd k8s-prometheus

RBAC authorization

kubectl apply -f prometheus-rbac.yaml

Configuration

A ConfigMap prometheus-config defines prometheus.yml with scrape jobs for the Prometheus server itself, Kubernetes nodes, API servers, cAdvisor, node‑exporter and kube‑state‑metrics.

StatefulSet deployment

kubectl apply -f prometheus-statefulset.yaml

Service exposure

kubectl apply -f prometheus-service.yaml

The service uses a fixed NodePort (30090) for external access.

Deploy Grafana

StatefulSet

kubectl apply -f grafana.yaml

Access

Grafana is reachable at http://<NodeIP>:30091 with default credentials admin / admin .

Visualizing Prometheus data in Grafana

Import community dashboards (e.g., dashboard 3119 for cluster resources) and adjust the PromQL queries as needed.

Deploy Alertmanager

Configuration

kubectl apply -f alertmanager-configmap.yaml
kubectl apply -f alertmanager-pvc.yaml
kubectl apply -f alertmanager-deployment.yaml
kubectl apply -f alertmanager-service.yaml

Alert rules are stored in prometheus-rules.yaml (e.g., InstanceDown, NodeFilesystemUsage, NodeMemoryUsage, NodeCPUUsage). Email notifications are configured via the Alertmanager config.

Testing alerts

Stopping kubelet on a node triggers the InstanceDown alert and sends an email notification.

After completing these steps you obtain a fully functional Prometheus‑Grafana‑Alertmanager monitoring stack for a Kubernetes cluster.