Why Prometheus’s TSDB Makes Massive Monitoring Feasible

Background

Many beginners feel intimidated by Prometheus because it introduces many concepts and seems high‑threshold. Key concepts include Instance, Job, Metric, Metric Name, Metric Label, Metric Value, Metric Type (Counter, Gauge, Histogram, Summary), DataType (Instant Vector, Range Vector, Scalar, String), Operator, and Function. Like Alibaba, Prometheus is fundamentally a data‑driven monitoring system.

Daily Monitoring

To monitor each API of a web server (e.g., WebServerA), dimensions such as service name (job), instance IP (instance), API name (handler), method, response code (code), and request count (value) are needed.

Example SQL‑like queries:

SELECT * from http_requests_total WHERE code="200" AND method="put" AND created_at BETWEEN 1495435700 AND 1495435710;

SELECT * from http_requests_total WHERE handler="prometheus" AND method="post" AND created_at BETWEEN 1495435700 AND 1495435710;

SELECT * from http_requests_total WHERE handler="query" AND instance="10.59.8.110" AND created_at BETWEEN 1495435700 AND 1495435710;

Monitoring 100 services with 10 instances each, 20 APIs, 4 methods, collecting every 30 seconds for 60 days yields 13.824 billion data points, which is impractical for relational databases; thus Prometheus uses a TSDB.

Storage Engine

TSDB perfectly fits monitoring data characteristics:

Massive data volume

Predominantly write operations

Writes are almost sequential, ordered by time

Writes rarely modify old data; data is written shortly after collection

Deletion occurs by block removal, not individual points

Data size typically exceeds memory, making caching ineffective

Reads are sequential (ascending or descending)

High‑concurrency reads are common

TSDB stores data as time‑series documents keyed by series ID and builds three indexes: Series, Label Index, and Time Index.

{"labels":[{"latency":"500"}],"samples":[{"timestamp":1473305798,"value":0.9}]}

Labels define dimensions; samples hold timestamp and value.

series
^
│ ... server{latency="500"}
│ ... server{latency="300"}
│ ... server{}
...
v
<-------- time -------->

TSDB uses timeseries:doc:: as the key for values and creates indexes to accelerate label‑and‑time range queries.

Data Computation

The robust storage engine enables powerful calculations. Prometheus can select metric series and apply operators and functions to perform matrix operations, as illustrated below.

This capability makes Prometheus comparable to a combined data‑warehouse and compute platform, pointing to the future direction of monitoring.

One Calculation, Many Queries

Because such computation is resource‑intensive, pre‑computing results (recording rules) is recommended. Frequently used or heavy expressions are evaluated once and stored as new time series, enabling fast repeated queries for dashboards and alerting rules.