How to Monitor Kubernetes with Prometheus, Grafana, and Helm – A Step‑by‑Step Guide

Why Use Prometheus for Kubernetes Monitoring

Prometheus is highly flexible and scalable, making it suitable for large‑scale Kubernetes clusters. It allows custom metrics and can monitor many applications and services. Its pull‑based data collection fits the dynamic lifecycle of containers and pods, eliminating the need for agents to push data.

Practical Scenario and Step‑by‑Step Implementation

1. Install and Configure Prometheus

Deploy the Prometheus Operator using Helm, which manages Prometheus instances via a custom resource definition (CRD).

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus

2. Define Targets and Metrics

Create a ConfigMap named prometheus-k8s-config that contains prometheus.yml to specify scrape jobs for nodes and pods.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-k8s-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'kubernetes-nodes'
        kubernetes_sd_configs:
          - role: node
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
          - role: pod

3. Create Alerting Rules

Define a PrometheusRule resource to trigger alerts such as high CPU usage.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
  name: example
spec:
  groups:
  - name: example
    rules:
    - alert: HighCpuUsage
      expr: 100 - (avg by (instance) (irate(node_cpu{mode="idle"}[5m])) * 100) > 90
      for: 5m
      labels:
        severity: warning
      annotations:
        description: 'High CPU usage on instance {{ $labels.instance }}'

4. Deploy Alertmanager

Install Alertmanager with Helm to handle alert notifications.

helm install alertmanager prometheus-community/alertmanager

5. Visualize Metrics with Grafana

Install Grafana via Helm and connect it to Prometheus for richer dashboards.

helm install grafana prometheus-community/grafana

6. Test the Monitoring Setup

Generate load against a service using ab and verify that alerts fire and appear in Grafana and email.

# ab -c 1000 -n 1000000 http://192.168.40.180:80/

After the test, the Grafana UI shows increased traffic, and the configured email address receives an alert notification.