How to Deploy Prometheus on Kubernetes and Resolve Alertmanager Port Issues

Prometheus Role

Prometheus Monitoring is an open‑source system monitoring and alerting tool originally developed by SoundCloud in 2012 and joined the Cloud Native Computing Foundation in 2016. It collects, stores, and queries metric data to help users monitor application and system performance.

Deployment Process

This article uses Prometheus to monitor a Kubernetes cluster and solves the Alertmanager port 9093 connection‑refused issue.

1. Download matching Prometheus version

# My k8s cluster version is 1.26.9, so I download v0.13
wget https://mirror.ghproxy.com/https://github.com/prometheus-operator/kube-prometheus/archive/refs/tags/v0.13.0.zip

# After download, unzip to use
unzip v0.13.0.zip

2. Customize alert rules and email push

cd kube-prometheus-0.13.0/manifests/
# Edit alert rule file
vim prometheus-prometheusRule.yaml
# Edit alert push file
vim alertmanager-secret.yaml

3. Deploy and delete Prometheus

# Deploy Prometheus
kubectl apply --server-side -f manifests/setup -f manifests
# Remove Prometheus
kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup
# After deployment, normal resource status appears

4. Adjust Service Types and Network Policies (if no Ingress)

# Change Service type to NodePort for external access
kubectl -n monitoring edit svc alertmanager-main
kubectl -n monitoring edit svc prometheus-k8s
kubectl -n monitoring edit svc grafana
# Delete default network policies that block traffic
kubectl -n monitoring delete networkpolicy --all

5. Issue with Alertmanager

# Describe the failing pod
kubectl -n monitoring describe pod alertmanager-main-1
# Error observed
dial tcp 10.244.135.151:9093 connection refused

# Edit Alertmanager main resource
kubectl -n monitoring edit alertmanager main
# Delete and recreate the resource after modification
kubectl -n monitoring delete alertmanager main
kubectl -n monitoring apply -f dump-modify.yaml

6. DNS and CNI Issues

# Check logs for DNS errors
kubectl -n monitoring logs sts alertmanager
# List all pods for context
kubectl get pod -A -o wide
# Delete problematic coredns pod
kubectl -n kube-system delete pod coredns-5bbd96d687-gtl9r
# Remove CNI configuration files that conflict
ls -l /etc/cni/net.d/

Summary

# Regardless of the service you deploy, if pods run and can communicate across nodes, it is not a network issue. For isolated pod failures, examine error messages; for port‑refused errors, first check the cluster's coredns configuration. Testing on a single‑node cluster can also aid debugging.