Understanding Prometheus Relabeling: Rules, Actions, and Practical Use Cases

Overview

Prometheus discovers, scrapes, and processes label objects; relabeling is used to transform and filter these labels. Common use cases include monitoring only targets with specific service‑discovery annotations, adding HTTP query parameters to scrape requests, storing only a subset of samples, and merging two label values into one.

Hidden Labels and Metadata

Labels that start with __ are special hidden labels that are removed after relabeling. They provide extra metadata about the target. Examples include __address__ (the target address, default : ), __scheme__ (http or https, default http), __metrics_path__ (default /metrics ), and __param_<name> (HTTP query parameters). These labels can be set or overridden via relabeling rules.

Service‑discovery mechanisms can also provide labels prefixed with __meta_ , such as __meta_kubernetes_pod_name and __meta_kubernetes_pod_ready , which expose Kubernetes metadata to Prometheus.

If a relabeling step needs a temporary label for later processing, the __tmp prefix can be used; such temporary labels are ignored by Prometheus itself.

Relabeling Rules

Each relabeling rule is a relabel_config block composed of a list of fields that are applied sequentially to each label set. The main fields are:

action : the relabeling action (replace, keep, drop, hashmod, labelmap, labeldrop, labelkeep). Default is replace .

separator : string used to join source_labels (default ; ).

source_labels : list of label names to be concatenated.

target_label : label to write to when using replace or hashmod .

regex : regular expression applied to the concatenated string (default (.*) ).

modulus : modulus value for hashmod (used for sharding).

replacement : string written to target_label , can reference captured groups.

Setting or Replacing Label Values

The replace action is used to set a fixed label value or to modify a label based on a regular expression. If action is omitted, replace is assumed.

action: replace
source_labels: [<source label name list>]
separator: <source labels separator>  # default ';'
regex: <regular expression>            # default '(.*)'
replacement: <replacement string>    # default '$1'
target_label: <target label>

Example: set a constant label env=production :

action: replace
replacement: production
target_label: env

Example: rewrite the port part of __address__ to always use port 80:

action: replace
source_labels: [__address__]
regex: ([^:]+)(?::\d+)?
replacement: "$1:80"
target_label: __address__

Keeping or Dropping Objects

The keep and drop actions filter entire label sets based on whether the concatenated source_labels match a regular expression.

action: keep
source_labels: [<source label list>]
separator: <separator>  # default ';'
regex: <regular expression>  # default '(.*)'

When the regex does not match, the object is removed (for keep ) or retained (for drop ).

Example: keep only targets that have the annotation example.io/should_be_scraped=true in Kubernetes:

action: keep
source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
regex: true

Example: keep only metrics whose name starts with api_ or http_ :

action: keep
source_labels: [__name__]
regex: "(api_|http_).*"

Label Mapping

The labelmap action remaps label names (not values) using a regular expression. It is often used to copy hidden Kubernetes service labels to new labels with a k8s_ prefix.

action: labelmap
regex: __meta_kubernetes_service_label_(.+)
replacement: "k8s_$1"

This copies the value of each __meta_kubernetes_service_label_<labelname> into a new label named k8s_<labelname> .

Keeping or Deleting Labels

The labelkeep action retains only labels whose names match the provided regex, while labeldrop removes matching labels.

action: labelkeep
regex: <regular expression>  # default '(.*)'

Example: remove the replica label from alerts generated by HA Prometheus instances:

action: labeldrop
regex: replica

Example: drop any label that starts with info_ :

action: labeldrop
regex: info_.*

Label Hashing and Sharding

The hashmod action computes a hash of the concatenated source_labels , applies a modulus, and stores the result in target_label . This is useful for horizontally scaling Prometheus by sharding targets.

action: hashmod
source_labels: [<source label list>]
modulus: <modulus value>
target_label: <target label>

Typical workflow: compute a hash of the instance label, store it in a temporary label, then keep only the targets whose hash equals a specific shard number.

- action: hashmod
  source_labels: [instance]
  modulus: 10
  target_label: __tmp_hashmod
- action: keep
  source_labels: [__tmp_hashmod]
  regex: 2

By combining hashmod with keep , you can retain only the subset of targets belonging to a chosen shard.

Understanding relabeling equips you with powerful tools to manipulate labels, filter targets, and adapt Prometheus scraping behavior to a wide range of monitoring scenarios.