Master PromQL: From Basics to Advanced Query Techniques

Data Types

Prometheus defines four metric types: Gauge , Counter , Histogram , and Summary . The tutorial focuses on Gauge and Counter, as they represent the most common monitoring data.

Gauge

A Gauge holds the current value of a metric, such as memory usage or load average, which can increase or decrease over time.

Counter

A Counter is a monotonically increasing value, like total network packets received. Monitoring typically cares about the increment or rate rather than the absolute value.

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.206.0.16 netmask 255.255.240.0 broadcast 10.206.15.255
RX packets 457952401 bytes 125894899868 (117.2 GiB)
TX packets 518040495 bytes 276312546157 (257.3 GiB)

Time‑Series Data

PromQL queries time‑series data. Understanding series and data points is essential. The example below shows memory availability for five machines over the last hour.

Switching to Table view reveals the latest values for each machine at a specific timestamp.

Query Types

Expressions like mem_available_percent{app="clickhouse"} return an Instant Vector . Adding a range selector, e.g., mem_available_percent{app="clickhouse"}[1m], creates a Range Vector . The former is used for instant queries, the latter for range queries.

Selectors

Label matchers filter series. Supported operators include = (exact match), != (not equal), =~ (regex match), and !~ (regex not match).

{__name__="mem_available_percent", app="clickhouse"}

Offset

The offset keyword fetches historical data, useful for week‑over‑week comparisons.

sum(http_requests_total{method="GET"} offset 1d)

Operators

Arithmetic

Supported symbols: +, -, *, /, %, ^. Example: compute memory availability from raw metrics.

(mem_available{app="clickhouse"} / mem_total{app="clickhouse"}) * 100

Comparison

Operators like <, >= enable threshold‑based alerts, e.g., flag memory availability below 60%.

mem_available_percent{app="clickhouse"} < 60

Logical/Set

Keywords and, or, unless combine vectors. Example combining disk usage and size:

disk_used_percent{app="clickhouse"} > 70
and
disk_total{app="clickhouse"}/1024/1024/1024 < 500

Vector Matching

Use on or ignoring to control label matching, and group_left / group_right for one‑to‑many or many‑to‑one joins.

mysql_slave_status_slave_sql_running == 0
and ON (instance)
mysql_slave_status_master_server_id > 0

Aggregations

Functions like sum, avg, min, max, count, bottomk, topk, and quantile operate on instant vectors. Example:

avg(mem_available_percent{app="clickhouse"})
bottomk(2, mem_available_percent{app="clickhouse"})

Key Functions

absent_over_time : Detects missing data for alerting.

increase and rate : Compute increments and per‑second rates, handling extrapolation.

irate : More sensitive rate using the last two points.

histogram_quantile : Estimates quantiles from histogram buckets.

_over_time variants (e.g., avg_over_time) operate on range vectors.

MetricsQL extensions like count_gt_over_time simplify complex conditions.

increase(net_bytes_recv{interface="eth0"}[1m])
rate(net_bytes_recv{interface="eth0"}[1m]) == bool increase(net_bytes_recv{interface="eth0"}[1m])/60.0
histogram_quantile(0.9, rate(http_request_duration_seconds_bucket[10m]))
count_gt_over_time(interface_status[5m], 10) >= 3

Conclusion

The article consolidates essential PromQL concepts and practical patterns, enabling users to write effective queries, build alerts, and perform advanced analysis on Prometheus metrics.