How to Deploy and Troubleshoot Prometheus Monitoring on a Kubernetes Cluster

Prometheus role:

Prometheus Monitoring is an open‑source system monitoring and alerting tool originally developed by SoundCloud in 2012 and joined the Cloud Native Computing Foundation in 2016. It collects, stores, and queries metric data to help users observe application and system performance.

Deployment process:

This article uses Prometheus to monitor k8s cluster resources and resolves the Alertmanager 9093 port connection‑refused issue.

1. Download the Prometheus version matching the k8s cluster version

# My k8s cluster version is 1.26.9, so I download version 0.13
wget https://mirror.ghproxy.com/https://github.com/prometheus-operator/kube-prometheus/archive/refs/tags/v0.13.0.zip
# After download, unzip to use
unzip v0.13.0.zip

2. Enter the extracted directory and customize alert rules and email notifications

cd kube-prometheus-0.13.0/manifests/
# This file configures alert rules
vim prometheus-prometheusRule.yaml
# This file configures alert notifications
vim alertmanager-secret.yaml

3. Deploy Prometheus monitoring and removal

kubectl apply --server-side -f manifests/setup -f manifests
# Remove Prometheus
kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup
# After deployment, the resources should be in a normal state

# If no ingress is deployed, change the following svc files to NodePort for external access
kubectl -n monitoring edit svc alertmanager-main
kubectl -n monitoring edit svc prometheus-k8s
kubectl -n monitoring edit svc grafana
# Delete all network policies that restrict traffic
kubectl -n monitoring delete networkpolicy --all

4. Discuss the issues I previously encountered

# When deploying Prometheus, Alertmanager failed to start with the error:
kubectl -n monitoring describe pod alertmanager-main-1
# dial tcp 10.244.135.151:9093 connection refused

# Some GitHub issues suggested editing the StatefulSet (STS) managed by the Alertmanager CRD, but changes did not take effect.
# Editing the timeout did not help; finally I inspected the container ports and discovered the real problem.
kubectl -n monitoring edit alertmanager main
kubectl -n monitoring delete alertmanager main

# Export the Alertmanager resource, modify it, and re‑apply
kubectl -n monitoring get alertmanager main -o yaml > dump-modify.yaml
vim dump-modify.yaml
# (YAML content omitted for brevity)
# Delete the existing main resource
kubectl -n monitoring delete alertmanager main
# Re‑create the main resource
kubectl -n monitoring apply -f dump-modify.yaml

# Logs indicated DNS resolution problems; the cluster uses a high‑availability setup with Calico, but CoreDNS was using a different service CIDR (10.88.0.0/16).
kubectl -n monitoring logs sts alertmanager
kubectl get pod -A -o wide

# The CNI network component showed a flannel interface (cni0) instead of Calico's, causing the mismatch.
# Deleting the original CoreDNS and cleaning up CNI configs restored network connectivity.
ls -l /etc/cni/net.d/
# Delete both conflicting CNI files
kubectl -n kube-system delete pod coredns-5bbd96d687-gtl9r
kubectl -n kube-system get pod -o wide

Summary

# When any service is deployed, ensure pods can run and communicate across nodes; if only specific pods fail with port‑refused errors, check CoreDNS and CNI configurations first. Using a single‑node test cluster can also help isolate issues.