Mastering Prometheus Relabeling: Rules, Actions, and Real-World Use Cases

Overview

Relabeling is the method for configuring Prometheus metadata; it is the core mechanism for transforming and filtering label objects. This article explains how relabeling rules operate and how they can be applied in various scenarios.

Only monitor targets that have specific service‑discovery annotations.

Add HTTP query parameters to scrape requests.

Store only a subset of samples extracted from a target.

Merge two label values into a single label.

Relabeling is implemented as a series of transformation steps that can be placed in Prometheus configuration files to filter or modify label objects. It can be applied to the following types of label objects:

Discovered scrape targets ( relabel_configs)

Individual scraped samples ( metric_relabel_configs)

Alerts sent to Alertmanager ( alert_relabel_configs)

Samples written to remote storage ( write_relabel_configs)

All of these relabeling configuration blocks share the same type relabel_config. Each block consists of a list of rules that are applied sequentially to each label object.

A relabeling rule can retain or discard an object based on a regular‑expression match, modify its labels, or map an entire set of labels to another set. Once a relabeling step decides to drop an object, no further steps are executed for that object and it is removed from the output list.

Hidden Labels and Metadata

Labels that start with a double underscore __ are special hidden labels that are removed after relabeling. They are initially attached to provide extra metadata about the label object and can be used during the relabeling phase to modify objects.

For scraped metrics, some hidden labels control how a target should be scraped: __address__: the address of the target, originally provided by the service‑discovery mechanism as <host>:<port>. If the instance label is not explicitly set, Prometheus sets it to the value of __address__ after relabeling. __scheme__: the request scheme (http or https), default is http. __metrics_path__: the HTTP path used to collect metrics, default is /metrics. __param_<name>: HTTP query‑parameter name and its value.

Service‑discovery mechanisms can also provide a set of labels prefixed with __meta_, containing specific discovery metadata. For example, when discovering pods in a Kubernetes cluster, the Kubernetes service‑discovery engine adds labels such as __meta_kubernetes_pod_name (the pod name) and __meta_kubernetes_pod_ready (whether the pod is ready). See the official documentation for more details.

If a relabeling step needs to store a value temporarily for later steps, a label prefixed with __tmp can be used; Prometheus itself never consumes __tmp labels.

Relabeling Rule Attributes

Relabeling rules are mainly composed of the following configuration fields (only a subset is used for each action): action: the relabeling action to perform. Possible values are replace, keep, drop, hashmod, labelmap, labeldrop, or labelkeep. The default is replace. separator: a string used to join the values of source_labels. Default is ;. source_labels: a list of source label names whose values are concatenated (using the separator) and matched against regex. target_label: the label to set when using replace or hashmod. regex: regular expression applied to the concatenated source label string. Default is (.*), which matches any value. modulus: the modulus used by hashmod to limit the hash value. replacement: the string written to target_label. It may reference capture groups from regex.

Setting or Replacing a Label Value

The most common operation is to set or overwrite a label value using the replace action. If the action field is omitted, replace is assumed.

action: replace
source_labels: [<source label name list>]
separator: <source labels separator>  # default ';'
regex: <regular expression>  # default '(.*)'
replacement: <replacement string>  # default '$1'
target_label: <target label>

Example: set a fixed label value.

action: replace
replacement: production
target_label: env

This configuration sets the env label to production for every target.

Example: replace the port part of __address__ with a fixed port 80.

action: replace
source_labels: [__address__]
regex: ([^:]+)(?::\d+)?  # first capture group matches host, second matches port
replacement: "$1:80"
target_label: __address__

Keeping or Dropping Objects

The keep and drop actions filter label objects. They are useful for deciding which discovery targets to scrape, which samples to store, or which alerts to forward.

action: keep
source_labels: [<source label name list>]
separator: <source labels separator>  # default ';'
regex: <regular expression>  # default '(.*)'

The keep action retains objects whose concatenated source labels match the regex; otherwise the object is removed. The drop action does the opposite.

Example: scrape only Kubernetes services that have the annotation example.io/should_be_scraped=true.

action: keep
source_labels:
  [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
regex: true

Example: keep only metrics whose names start with api_ or http_.

action: keep
source_labels: [__name__]
regex: "(api_|http_).*"

Label Mapping

The labelmap action maps source label names to new label names. It is often used to copy hidden or temporary metadata labels from service discovery into more convenient target labels.

action: labelmap
regex: __meta_kubernetes_service_label_(.+)
replacement: "k8s_$1"

This configuration copies any label matching __meta_kubernetes_service_label_<labelname> to a new label named k8s_<labelname>.

Keeping or Dropping Specific Labels

The labelkeep and labeldrop actions allow selective retention or removal of individual labels.

action: labelkeep
regex: <regular expression>  # default '(.*)'

Example: drop the replica label from alerts generated by high‑availability Prometheus instances.

action: labeldrop
regex: replica

Example: drop all labels that start with info_ to reduce storage overhead.

action: labeldrop
regex: info_.*

Hashing and Sharding

The hashmod action computes a hash of concatenated source label values, applies a modulus, and stores the result in a target label. This is useful for horizontal scaling of Prometheus instances.

action: hashmod
source_labels: [<source label name list>]
modulus: <modulus value>
target_label: <target label>

Typical usage combines hashmod with keep to retain only a specific shard. Example: keep only targets whose hash modulo 10 equals 2.

- action: hashmod
  source_labels: [instance]
  modulus: 10
  target_label: __tmp_hashmod
- action: keep
  source_labels: [__tmp_hashmod]
  regex: 2

After applying hashmod, the hash value is stored in the temporary label __tmp_hashmod. The subsequent keep rule retains only those objects whose hash equals 2, achieving sharding.

With this knowledge, you now have a solid understanding of Prometheus relabeling and can explore service discovery integration in greater depth.