How to Build a Complete Kubernetes Monitoring Stack with Prometheus & Grafana

Monitoring Solution

cAdvisor + node_exporter + Prometheus + Grafana

cAdvisor – data collection

node-exporter – aggregation

Prometheus – processing and storage

Grafana – visualization

Monitoring Process

Container monitoring: Prometheus uses cAdvisor (integrated in K8s kubelet) to collect container metrics, stores them, and visualizes with Grafana.

Node monitoring: node_exporter collects host resources, Prometheus stores them, Grafana visualizes.

Master node monitoring: kube-state-metrics exposes apiserver data, Prometheus stores it, Grafana visualizes.

Kubernetes Monitoring Metrics

K8S Native Metrics

Node resource utilization – CPU, memory, disk.

Node count – evaluate cost and scaling.

Pod count – assess load and server requirements.

Resource object status – monitor pods, controllers, jobs, etc.

Pod Monitoring

Number of pods per project – healthy vs problematic.

Container resource utilization – CPU, network, memory.

Application metrics – concurrency, request/response performance.

Service Discovery

Continuously fetch targets from the K8s API to stay in sync with cluster state.

node – auto‑discover cluster nodes

pod – auto‑discover running containers and ports

service – auto‑discover Service IPs and ports

endpoints – auto‑discover containers within pods

ingress – auto‑discover ingress rules and entry points

Prometheus Monitoring Kubernetes Deployment Practice

Deployment Preparation

Images are pulled only on the master node.

Deploy node-exporter via DaemonSet

Deploy Prometheus

Deploy Grafana

Verification

1) Check pod/service information.

2) Open the Grafana page (http://10.10.11.202:30931, admin/admin) and import the K8S Dashboard template.

YAML Configuration Files

node-exporter.yaml

rbac-setup.yaml

configmap.yaml

prometheus.deploy.yml

prometheus.svc.yml

grafana-deploy.yaml

grafana-svc.yaml