How to Expose Ingress Metrics for Prometheus Monitoring in Kubernetes

Preface

Recently a client needed to monitor ingress traffic. After researching, the deployment was successful and the process and issues are recorded here.

Expose Ingress Monitoring Port

By default nginx‑ingress exposes metrics on port 10254 at path /metrics. To make the port reachable, edit the Service definition to add a listener named https-metrics on port 10254, and modify the deployment to open the same port in the pod as metrics.

spec:
  type: ClusterIP
  ports:
  - name: https-webhook
    port: 443
    targetPort: webhook
  - name: https-metrics
    port: 10254
    targetPort: 10254

ports:
- name: http
  containerPort: 80
  protocol: TCP
- name: https
  containerPort: 443
  protocol: TCP
- name: webhook
  containerPort: 8443
  protocol: TCP
- name: metrics
  containerPort: 10254
  protocol: TCP

Static Scrape Configuration

Add a custom scrape job via the additionalScrapeConfigs field in prometheus-prometheus.yaml. Create prometheus-additional.yaml with the following content:

- job_name: nginx-ingress
  metrics_path: /metrics
  scrape_interval: 5s
  static_configs:
  - targets:
    - 172.16.200.102:10254
    - 172.16.200.103:10254
    - 172.16.200.104:10254

Create a secret that contains this file:

$ kubectl create secret generic ingress-nginx-additional-configs --from-file=./prometheus-additional.yaml -n monitoring

Reference the secret in prometheus-prometheus.yaml:

serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.11.0
additionalScrapeConfigs:
  name: ingress-nginx-additional-configs
  key: prometheus-additional.yaml

Re‑apply the Prometheus configuration and verify the targets are up in the UI.

Import Grafana dashboard 9614 to visualise the metrics.

ServiceMonitor‑Based Scrape

When Prometheus is deployed via the Operator, use the ServiceMonitor CRD. Create a ServiceMonitor that selects the ingress‑nginx pods and points to the https-metrics port.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-ingress
  namespace: monitoring
  labels:
    app.kubernetes.io/component: controller
spec:
  jobLabel: app.kubernetes.io/component
  endpoints:
  - port: https-metrics
    interval: 10s
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
  namespaceSelector:
    matchNames:
    - ingress-nginx

Apply the resource and verify the new target appears in Prometheus.

References

https://www.amd5.cn/atang_4421.html

https://www.cnblogs.com/lvcisco/p/12574532.html

https://prometheus.io/docs/prometheus/latest/configuration/configuration