Mastering Prometheus: Architecture, Metrics, and Real-World Monitoring Techniques

1. Prometheus

Prometheus is an open‑source monitoring and alerting toolkit that stores collected metrics as time‑series data, each consisting of a timestamp, a value, and optional key‑value pair labels. It is widely used to monitor Kubernetes clusters.

1.1 Suitable and Unsuitable Scenarios

Suitable : Recording any numeric time series, machine‑centric monitoring, highly dynamic service‑oriented architectures, multi‑dimensional data queries, reliability‑focused environments, and standalone operation without external storage.

Unsuitable : Situations requiring 100% accuracy such as per‑request billing, where other systems are preferable.

1.2 Data Model

Prometheus stores data locally in a TSDB (time‑series database). The core design revolves around blocks and a Write‑Ahead Log (WAL). Each block contains chunks, an index, meta.json, and tombstones. Blocks are identified by a ULID, enabling easy chronological sorting.

The WAL ensures that data residing in memory is persisted to disk before a crash, allowing recovery by replaying WAL entries into the head block.

1.3 Metric Types

Counter : Monotonically increasing values (e.g., total packets received).

Gauge : Values that can go up or down (e.g., temperature, memory usage).

Histogram : Aggregates observations into configurable buckets, useful for percentiles and request counts.

1.4 Metric Summarization and Aggregation

Single metrics often have limited value; aggregating and visualizing multiple metrics provides insight. Common aggregation functions include count, sum, average, median, percentiles, standard deviation, and rate.

1.5 Exporters

Prometheus uses exporter programs to expose metrics from hosts and applications. Notable exporters include Node Exporter for system metrics and cAdvisor for Docker container statistics.

1.6 Target Lifecycle

Service discovery → configuration → relabeling → scraping → metrics relabeling.

1.7 PromQL Query Language

PromQL provides selectors, label matchers, range vectors, offsets, sub‑queries, and operators.

Selectors : Define a set of label matchers and a metric name, e.g., {job="node"}.

Label Matchers : Use =, !=, =~, !~ to filter series.

Range Vectors : Append [5m] to a selector to select data over a time window.

Offset : offset 1h queries data from one hour ago.

Sub‑queries : Nest a range selector inside another query.

Common functions include label_join(), label_replace(), predict_linear(), rate(), irate(), sort(), and sort_desc().

1.8 Practical Queries

CPU Usage

// avg(irate(node_cpu_seconds_total{job="node"}[5m] by (instance) * 100))

CPU Load (Saturation)

Compare node_load1 against twice the number of CPU cores:

count by (instance)(node_cpu_seconds_total{mode="idle"})
node_load1 > on (instance) 2 * count by (instance)(node_cpu_seconds_total{mode="idle"})

Memory Usage

Memory metrics are prefixed with node_memory_. Example calculation:

// Total memory
node_memory_MemTotal_bytes
// Free memory
node_memory_MemFree_bytes
// Buffers
node_memory_Buffers_bytes
// Cached
node_memory_Cached_bytes
// Usage %
(node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Buffers_bytes - node_memory_Cached_bytes) / node_memory_MemTotal_bytes * 100

Memory Saturation

Use node_vmstat_pswpin and node_vmstat_pswpout (bytes swapped per second) to compute swap activity:

1024 * sum by (instance) (rate(node_vmstat_pgpgin[1m]) + rate(node_vmstat_pgpgout[1m]))

Disk Usage

Calculate filesystem usage percentage:

(node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_free_bytes{mountpoint="/"}) / node_filesystem_size_bytes{mountpoint="/"} * 100

Predict disk exhaustion with predict_linear:

predict_linear(node_filesystem_free_bytes{mountpoint="/"}[1h], 4*3600) < 0