Mastering Prometheus: From Metrics Collection to Alerting and Visualization

Monitoring Modes

There are two ways to collect metrics: push (e.g., ElasticSearch, InfluxDB, OpenTSDB) where applications send data via TCP/UDP, and pull (e.g., Prometheus) where the monitoring system scrapes metrics endpoints, avoiding dependence on the monitor's availability.

Push can cause back‑pressure on the application if the monitor fails, while UDP may lose data. Pull models like Prometheus use service discovery (DNS‑SRV, Consul) to add targets automatically. InfluxDB with a collector or Elasticsearch with metricbeat can act as pull, and Prometheus with Pushgateway can act as push.

Prometheus Overview

Prometheus, named after the Greek god of foresight, is inspired by Google’s internal monitoring system Borgmon. It scrapes metrics from configured targets or via service discovery and stores them locally on disk. Exporters expose metrics for services such as databases or load balancers.

The collected data is used for alerting and visualization .

Metrics Format

Metrics follow a simple text‑based format:

<metric_name>{<label_name>=<label_value>, ...} <value>

Rules:

Metric name: [a-zA-Z_:][a-zA-Z0-9_:]* Label name: [a-zA-Z0-9_]* Label value: any string (prefer enumerated values, avoid high‑cardinality data like user IDs or emails).

Implementing Metrics in Node.js

The prom-client library provides a full‑featured client. npm install prom-client --save Example of collecting default Node.js metrics:

const promClient = require('prom-client');
promClient.collectDefaultMetrics({
  timeout: 5000,
});

Custom metrics can be created: Counter – e.g., number of user registrations or revenue. Gauge – e.g., current memory usage or CPU load. Histogram or Summary – for request latency, with Histogram giving bucketed counts and Summary providing quantiles.

Alerting with Alertmanager

Prometheus’s Alertmanager handles deduplication, grouping, inhibition, and silencing of alerts, reducing noise from repeated or low‑priority notifications. Alerts can be routed via webhook, email, Slack, etc.

Grouping : combine many similar alerts into a single notification.

Inhibition : suppress alerts that are less important when a higher‑priority alert is firing.

Silencing : temporarily mute alerts that are not relevant.

Visualization with Grafana

Grafana is the preferred UI for visualizing Prometheus data. After adding Prometheus as a data source, you can create dashboards using PromQL queries. Useful query helpers include: label_values(label) – all values of a label. label_values(metric, label) – values of a label for a specific metric. metrics(metric) – regex‑matched metric names. query_result(query) – raw query results.

References

https://prometheus.io/docs/operating/configuration/

https://prometheus.io/blog/2015/06/01/advanced-service-discovery/

http://docs.grafana.org/features/datasources/prometheus/