How to Identify Unused Prometheus Metrics with Mimirtool and Reduce Resource Usage

Introduction

The author needed to track and resolve series cardinality issues while significantly reducing Prometheus resource consumption. To achieve this, they first analyze the system and explain how to use mimirtool to identify which metrics are used on the platform and which are not.

Prerequisites

The steps assume a Kubernetes cluster running the kube-prometheus-stack. Adjustments may be required for other deployment methods, but any setup with at least one Prometheus or Grafana instance should work.

Analyzing Prometheus Metrics Usage

The first task is to determine which metrics are in use. Previously this was done manually with grep, but mimirtool simplifies the process.

Metrics in Grafana Dashboards

To extract the list of metrics used by Grafana, create an API key with admin role (via http://grafana.your.domain/org/apikeys) and forward the Grafana pod port:

# Archlinux
pacman -Sy kubectl mimir jq
# MacOS
brew install kubectl mimirtool jq

# kubectl port-forward ${my_grafana_pod} -n monitoring 3000:3000

Open http://localhost:3000/org/apikeys, click New API key , set a name, choose Admin role, and copy the token: GRAFANA_API_TOKEN="copy your token here" Run the analysis:

mimirtool analyze grafana --address=http://localhost:3000 --key="${GRAFANA_API_TOKEN}"

The command creates metrics-in-grafana.json containing the JSON list of metrics referenced by Grafana dashboards.