Master Prometheus PQL: Essential Queries, Functions, and Tips

Overview

Prometheus is a Go‑written monitoring system that focuses on basic monitoring and retains 15 days of data by default, which is sufficient for most operational troubleshooting. It offers a dedicated query language called PQL that enables multi‑dimensional, function‑based analysis of collected metrics, providing highly customizable metric queries. This article walks through the essential PQL syntax and records the learning process.

PQL Important Concepts

Instant Vector

An instant vector represents the value of a metric at a single point in time.

node_cpu_seconds_total{mode="idle"}

Range Vector

A range vector contains metric values over a specified time interval.

node_cpu_seconds_total{mode="idle"}[5m]

Metric Types

Gauge : a value that can go up and down, e.g., CPU usage.

Counter : a cumulative value that only increases, e.g., process uptime.

Labels

Metrics can have multiple labels that describe additional dimensions. Labels can be filtered with regular expressions, similar to SQL WHERE clauses. node_cpu_seconds_total{mode="idle"} When a label selector is applied, only series matching the selector are returned, analogous to SELECT * FROM node_cpu_seconds_total WHERE mode='idle'.

Offset

The offset keyword queries data relative to the current time, e.g., node_cpu_seconds_total offset 5m retrieves the metric value from five minutes ago.

Comments

PQL uses # for comments.

Functions

Aggregation Functions (Instant Vectors)

sum : sums all values, e.g., sum(node_cpu_seconds_total).

min : returns the minimum value, e.g., min(node_cpu_seconds_total).

max : returns the maximum value, e.g., max(node_cpu_seconds_total).

avg : calculates the average, e.g., avg(node_cpu_seconds_total).

count : counts the number of series, e.g., count(node_cpu_seconds_total).

topk : returns the top N series, e.g., topk(5, node_cpu_seconds_total).

bottomk : returns the bottom N series.

Arithmetic Operations

Vectors can be combined with +, -, *, /. Operations are performed only on series that share identical label sets unless ignoring or group_left/right modifiers are used. node_cpu_seconds_total + node_cpu_seconds_total If the label sets differ, the result may be nodata unless ignored.

Logical Comparisons

==

and != for equality. >=, <=, >, < for numeric comparisons.

Set Operators

and : returns series that share common labels.

or : merges series, keeping one copy of duplicate labels.

unless : returns series from the left operand that have no matching series on the right.

Instant‑Vector Functions

abs : absolute value.

absent : returns 1 if a series does not exist.

ceil / floor : round up or down to the nearest integer.

clamp_max / clamp_min : limit values to a maximum or minimum.

Range‑Vector Functions

absent_over_time : detects missing data in a range.

changes : counts how many elements changed within the range.

delta : difference between first and last sample (requires gauge).

deriv : per‑second linear rate (requires gauge).

idelta , irate : similar to delta and deriv but handle irregular intervals.

<aggregation>_over_time : applies an aggregation (sum, avg, min, max, count) over the range.

Example Queries

select * from node_cpu_seconds_total where mode = 'idle'

select * from node_cpu_seconds_total where mode = 'idle' or mode = 'iowait'

select * from node_cpu_seconds_total where mode = 'idle' and cpu = '0'

Visual Aids