How to Expose Ingress Metrics and Scrape Them with Prometheus in Kubernetes

Expose Ingress Monitoring Port

By default the nginx‑ingress controller exposes metrics on port 10254 at the /metrics path. To make the port reachable, add a Service entry that listens on 10254 and name it https-metrics. Then modify the Deployment to open the pod port 10254 and name it metrics.

spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
    - name: https-metrics
      port: 10254
      targetPort: 10254

ports:
  - name: http
    containerPort: 80
    protocol: TCP
  - name: https
    containerPort: 443
    protocol: TCP
  - name: webhook
    containerPort: 8443
    protocol: TCP
  - name: metrics
    containerPort: 10254
    protocol: TCP

Static Scrape Configuration

In the prometheus-prometheus.yaml file, use the built‑in additionalScrapeConfigs field to add custom scrape targets. Create a separate file prometheus-additional.yaml with the following content:

- job_name: nginx-ingress
  metrics_path: /metrics
  scrape_interval: 5s
  static_configs:
    - targets:
      - 172.16.200.102:10254
      - 172.16.200.103:10254
      - 172.16.200.104:10254

Create a secret that contains this file:

$ kubectl create secret generic ingress-nginx-additional-configs --from-file=./prometheus-additional.yaml -n monitoring

Reference the secret in prometheus-prometheus.yaml under additionalScrapeConfigs:

serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.11.0
additionalScrapeConfigs:
  name: ingress-nginx-additional-configs
  key: prometheus-additional.yaml

Re‑apply the Prometheus configuration and verify that the new targets appear as up in the Prometheus UI.

Import Grafana dashboard 9614 to visualise the metrics; the data should display correctly.

Dynamic Scrape with ServiceMonitor

When Prometheus is deployed via the Operator, you can use the ServiceMonitor CRD ( servicemonitors.monitoring.coreos.com) to automatically discover ingress‑nginx pods. Create a file ingress-nginx.yaml with the following definition:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-ingress
  namespace: monitoring
  labels:
    app.kubernetes.io/component: controller
spec:
  jobLabel: app.kubernetes.io/component
  endpoints:
    - port: https-metrics
      interval: 10s
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
  namespaceSelector:
    matchNames:
      - ingress-nginx

Apply the resource:

$ kubectl apply -f ./ingress-nginx.yaml

After applying, the new ServiceMonitor appears in the Prometheus /targets page.

Finally, import the same Grafana dashboard as before to visualise the dynamically scraped metrics.

References

AMD5 tutorial

CNBlogs example

Prometheus documentation