Mastering Prometheus Relabeling: Rules, Actions, and Real-World Use Cases

Overview

Relabeling is the method for configuring Prometheus metadata; it is the core mechanism for transforming and filtering label objects. This article explains how relabeling rules operate and how they can be applied in various scenarios.

Only monitor targets that have specific service‑discovery annotations.

Add HTTP query parameters to scrape requests.

Store only a subset of samples extracted from a target.

Merge two label values into a single label.

Relabeling is implemented as a series of transformation steps that can be placed in Prometheus configuration files to filter or modify label objects. It can be applied to the following types of label objects:

Discovered scrape targets (

relabel_configs

)

Individual scraped samples (

metric_relabel_configs

)

Alerts sent to Alertmanager (

alert_relabel_configs

)

Samples written to remote storage (

write_relabel_configs

)

All of these relabeling configuration blocks share the same type

relabel_config

. Each block consists of a list of rules that are applied sequentially to each label object.

A relabeling rule can retain or discard an object based on a regular‑expression match, modify its labels, or map an entire set of labels to another set. Once a relabeling step decides to drop an object, no further steps are executed for that object and it is removed from the output list.

Hidden Labels and Metadata

Labels that start with a double underscore

__

are special hidden labels that are removed after relabeling. They are initially attached to provide extra metadata about the label object and can be used during the relabeling phase to modify objects.

For scraped metrics, some hidden labels control how a target should be scraped:

__address__

: the address of the target, originally provided by the service‑discovery mechanism as

<host>:<port>

. If the

instance

label is not explicitly set, Prometheus sets it to the value of

__address__

after relabeling.

__scheme__

: the request scheme (http or https), default is http.

__metrics_path__

: the HTTP path used to collect metrics, default is

/metrics

.

__param_<name>

: HTTP query‑parameter name and its value.

Service‑discovery mechanisms can also provide a set of labels prefixed with

__meta_

, containing specific discovery metadata. For example, when discovering pods in a Kubernetes cluster, the Kubernetes service‑discovery engine adds labels such as

__meta_kubernetes_pod_name

(the pod name) and

__meta_kubernetes_pod_ready

(whether the pod is ready). See the official documentation for more details.

If a relabeling step needs to store a value temporarily for later steps, a label prefixed with

__tmp

can be used; Prometheus itself never consumes

__tmp

labels.

Relabeling Rule Attributes

Relabeling rules are mainly composed of the following configuration fields (only a subset is used for each action):

action

: the relabeling action to perform. Possible values are

replace

,

keep

,

drop

,

hashmod

,

labelmap

,

labeldrop

, or

labelkeep

. The default is

replace

.

separator

: a string used to join the values of

source_labels

. Default is

;

.

source_labels

: a list of source label names whose values are concatenated (using the separator) and matched against

regex

.

target_label

: the label to set when using

replace

or

hashmod

.

regex

: regular expression applied to the concatenated source label string. Default is

(.*)

, which matches any value.

modulus

: the modulus used by

hashmod

to limit the hash value.

replacement

: the string written to

target_label

. It may reference capture groups from

regex

.

Setting or Replacing a Label Value

The most common operation is to set or overwrite a label value using the

replace

action. If the

action

field is omitted,

replace

is assumed.

<code>action: replace
source_labels: [&lt;source label name list&gt;]
separator: &lt;source labels separator&gt;  # default ';'
regex: &lt;regular expression&gt;  # default '(.*)'
replacement: &lt;replacement string&gt;  # default '$1'
target_label: &lt;target label&gt;
</code>

Example: set a fixed label value.

<code>action: replace
replacement: production
target_label: env
</code>

This configuration sets the

env

label to

production

for every target.

Example: replace the port part of

__address__

with a fixed port 80.

<code>action: replace
source_labels: [__address__]
regex: ([^:]+)(?::\d+)?  # first capture group matches host, second matches port
replacement: "$1:80"
target_label: __address__
</code>

Keeping or Dropping Objects

The

keep

and

drop

actions filter label objects. They are useful for deciding which discovery targets to scrape, which samples to store, or which alerts to forward.

<code>action: keep
source_labels: [&lt;source label name list&gt;]
separator: &lt;source labels separator&gt;  # default ';'
regex: &lt;regular expression&gt;  # default '(.*)'
</code>

The

keep

action retains objects whose concatenated source labels match the regex; otherwise the object is removed. The

drop

action does the opposite.

Example: scrape only Kubernetes services that have the annotation

example.io/should_be_scraped=true

.

<code>action: keep
source_labels:
  [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
regex: true
</code>

Example: keep only metrics whose names start with

api_

or

http_

.

<code>action: keep
source_labels: [__name__]
regex: "(api_|http_).*"
</code>

Label Mapping

The

labelmap

action maps source label names to new label names. It is often used to copy hidden or temporary metadata labels from service discovery into more convenient target labels.

<code>action: labelmap
regex: __meta_kubernetes_service_label_(.+)
replacement: "k8s_$1"
</code>

This configuration copies any label matching

__meta_kubernetes_service_label_<labelname>

to a new label named

k8s_<labelname>

.

Keeping or Dropping Specific Labels

The

labelkeep

and

labeldrop

actions allow selective retention or removal of individual labels.

<code>action: labelkeep
regex: &lt;regular expression&gt;  # default '(.*)'
</code>

Example: drop the

replica

label from alerts generated by high‑availability Prometheus instances.

<code>action: labeldrop
regex: replica
</code>

Example: drop all labels that start with

info_

to reduce storage overhead.

<code>action: labeldrop
regex: info_.*
</code>

Hashing and Sharding

The

hashmod

action computes a hash of concatenated source label values, applies a modulus, and stores the result in a target label. This is useful for horizontal scaling of Prometheus instances.

<code>action: hashmod
source_labels: [&lt;source label name list&gt;]
modulus: &lt;modulus value&gt;
target_label: &lt;target label&gt;
</code>

Typical usage combines

hashmod

with

keep

to retain only a specific shard. Example: keep only targets whose hash modulo 10 equals 2.

<code>- action: hashmod
  source_labels: [instance]
  modulus: 10
  target_label: __tmp_hashmod
- action: keep
  source_labels: [__tmp_hashmod]
  regex: 2
</code>

After applying

hashmod

, the hash value is stored in the temporary label

__tmp_hashmod

. The subsequent

keep

rule retains only those objects whose hash equals 2, achieving sharding.

With this knowledge, you now have a solid understanding of Prometheus relabeling and can explore service discovery integration in greater depth.