Deploy a Complete Prometheus Monitoring Stack on Kubernetes (Step‑by‑Step)

Introduction

Prometheus is an open‑source monitoring system originally built at SoundCloud and now a CNCF project, making it a natural fit for Kubernetes monitoring.

Prometheus Architecture

Prometheus consists of a server, client libraries, a push gateway, exporters, Alertmanager, and a simple web UI.

All data are stored as time‑series identified by a metric name and a set of key‑value labels.

Jobs and Instances

Instances are scrape targets; a job groups instances that share the same scrape configuration.

Kubernetes Monitoring Metrics

Important metrics include node resource utilization, pod counts, and resource‑object status. Implementations use cAdvisor for pod metrics, node‑exporter for node metrics, and kube‑state‑metrics for object status.

Deploy Prometheus on the K8s Cluster

Cluster environment

Four nodes are used: one master (192.168.73.138) and three workers (192.168.73.136, .139, .140).

Clone the project

<code>git clone https://github.com/zhangdongdong7/k8s-prometheus.git
cd k8s-prometheus</code>

RBAC authorization

<code>kubectl apply -f prometheus-rbac.yaml</code>

Configuration

A ConfigMap

prometheus-config

defines

prometheus.yml

with scrape jobs for the Prometheus server itself, Kubernetes nodes, API servers, cAdvisor, node‑exporter and kube‑state‑metrics.

StatefulSet deployment

<code>kubectl apply -f prometheus-statefulset.yaml</code>

Service exposure

<code>kubectl apply -f prometheus-service.yaml</code>

The service uses a fixed NodePort (30090) for external access.

Deploy Grafana

StatefulSet

<code>kubectl apply -f grafana.yaml</code>

Access

Grafana is reachable at

http://<NodeIP>:30091

with default credentials admin / admin .

Visualizing Prometheus data in Grafana

Import community dashboards (e.g., dashboard 3119 for cluster resources) and adjust the PromQL queries as needed.

Deploy Alertmanager

Configuration

<code>kubectl apply -f alertmanager-configmap.yaml
kubectl apply -f alertmanager-pvc.yaml
kubectl apply -f alertmanager-deployment.yaml
kubectl apply -f alertmanager-service.yaml</code>

Alert rules are stored in

prometheus-rules.yaml

(e.g., InstanceDown, NodeFilesystemUsage, NodeMemoryUsage, NodeCPUUsage). Email notifications are configured via the Alertmanager config.

Testing alerts

Stopping

kubelet

on a node triggers the

InstanceDown

alert and sends an email notification.

After completing these steps you obtain a fully functional Prometheus‑Grafana‑Alertmanager monitoring stack for a Kubernetes cluster.