From Single‑Node to Distributed Monitoring Storage: Scaling TSDB for Million‑QPS

Why the Starting Point Is a Single Machine

Single‑node TSDBs like Prometheus are simple, low‑latency, and cheap, making them the obvious choice for small‑scale monitoring. A Prometheus instance runs as a single binary, stores data locally on SSD, and can be inspected via Grafana with just a systemd service and a config file.

The storage model splits data into three layers: an in‑memory head block with a write‑ahead log (WAL), immutable blocks flushed to disk after a default two‑hour interval, and compaction of older blocks. Gorilla compression (delta‑of‑delta timestamps and XOR‑encoded values) reduces a 16‑byte sample to about 1.3 bytes, allowing billions of samples on a single disk.

The Five Walls That Hit a Single‑Node TSDB

1. Capacity limit – Disk size caps total data, and memory limits the number of active series; with millions of series the head alone can consume dozens of gigabytes.

2. Single‑point failure – If the machine crashes, all monitoring data is lost and the whole system goes blind, often at the worst moment (e.g., during a traffic surge).

3. Short retention – Prometheus keeps only 15 days by default, which is insufficient for quarterly capacity planning, compliance audits, or historical incident analysis.

4. No global view – Scaling out by deploying multiple Prometheus instances creates data islands; a cross‑region dashboard cannot be built because each instance only sees its own data.

5. Write and query throughput limits – A single node cannot sustain millions of samples per second or handle large‑scale PromQL queries without OOM failures.

First Evolution Step: Remote Write Decouples Scrape and Storage

After hitting the walls, the immediate reaction is to offload storage while keeping Prometheus as a lightweight collector. remote_write streams samples to an external, horizontally scalable backend, and remote_read allows queries to fetch historic data from that backend. Prometheus thus becomes a “scrape forwarder”.

Path 1 – Object Storage as a Cheap Long‑Term Layer (Thanos)

Thanos adds a thin overlay to existing Prometheus deployments:

Sidecar : runs alongside each Prometheus, continuously uploads local blocks to object storage and serves the most recent data for queries.

Store Gateway : reads blocks from object storage, translates PromQL into block scans, and caches indexes for faster access.

Compactor : merges small blocks into larger ones and performs down‑sampling (e.g., 15 s → 5 min → 1 h).

Query : fans out queries to all Sidecars and Store Gateways, merges results, and deduplicates samples from HA‑paired Prometheus instances.

This route requires minimal changes—just a few extra components—while providing near‑infinite long‑term storage, global querying, and HA deduplication. It is often the most cost‑effective solution for medium‑scale teams.

Path 2 – Native Horizontally Scalable Distributed TSDBs

When scale grows beyond what Thanos can comfortably handle, native distributed TSDBs such as Cortex, Grafana Mimir, VictoriaMetrics, and M3DB redesign the storage layer itself.

Key components:

Distributor : receives all remote_write samples, hashes series labels, and routes each series to a set of Ingester nodes via consistent hashing.

Ingester : stores data in a head + WAL just like Prometheus, but writes each series to multiple replicas (default factor 3) using quorum writes for HA.

Long‑term storage : flushed blocks are stored in object storage, identical to the Thanos path.

Query Frontend : splits large time‑range queries into smaller sub‑queries (e.g., per‑day) for parallel execution and caching.

Querier : fetches recent data from Ingesters and historic data from object storage, merges and deduplicates results.

Advantages over Thanos include true horizontal scaling, multi‑tenant isolation (per‑tenant series space and limits), and fine‑grained tiered storage with down‑sampling that keeps long‑term trends cheap.

Core Storage Engine Optimizations

All TSDBs share several low‑level techniques:

Compression encoding : Gorilla delta‑of‑delta timestamps and XOR value compression, combined with columnar layout and immutable chunks, achieve ~1.3 bytes per sample.

Inverted index : each label key‑value pair maintains a postings list; queries intersect postings to locate matching series, explaining why high‑cardinality labels can explode index size.

WAL & crash recovery : every write is first persisted to a WAL; on restart the WAL is replayed to rebuild the head.

Compaction : background merging of small blocks into larger ones reduces file count and enables additional compression and down‑sampling.

Real‑World Choices

Practitioners typically select from a spectrum:

Prometheus local storage – simple, single‑process, ideal for small clusters.

Thanos – Prometheus + object storage, incremental upgrade, best cost‑performance for medium scale.

Cortex / Grafana Mimir – native distributed, multi‑tenant, suited for very large platforms.

VictoriaMetrics – compact design (vmstorage/vminsert/vmselect) with aggressive compression.

M3DB – Uber’s in‑house distributed TSDB for extreme scale.

InfluxDB (IOx) – Arrow‑based columnar engine with object storage.

TimescaleDB – PostgreSQL‑based, adds SQL and transactional capabilities.

ClickHouse – columnar OLAP engine used by some teams for massive metric ingestion.

Distributed Is Not Free Lunch

Each added capability brings new costs:

Operational complexity : a single Prometheus process becomes a suite of 7‑8 components (Distributor, Ingester, Querier, Compactor, Store Gateway, etc.) that all need deployment, monitoring, scaling, and troubleshooting.

Consistency and deduplication : HA pair deduplication, quorum writes, and result merging are common sources of subtle bugs.

Query fan‑out latency : spreading a query across many nodes introduces tail latency; slow nodes can degrade overall response time.

Cost governance : object‑storage API calls, data scans, and cross‑region traffic can inflate bills if down‑sampling and retention policies are not carefully managed.

Therefore, the pragmatic conclusion is: most small‑ to medium‑size teams should stick with “single‑node Prometheus + Thanos for long‑term storage”. Only environments that truly demand millions of QPS, tens of thousands of instances, multi‑tenant isolation, and multi‑region deployment should adopt a native distributed TSDB.

Final Thoughts

The evolution from a single‑machine TSDB to a horizontally scalable, highly available, tiered‑storage, long‑term, globally queryable time‑series system follows a clear path: start with Prometheus, add remote_write + object storage (Thanos) when the five walls appear, and move to a native distributed TSDB when scale forces you beyond that.

Ask yourself: where does your current monitoring storage sit on this spectrum? Are you still hard‑pushing a single node, have you been surprised by object‑storage costs, or are you already wrestling with a seven‑component distributed stack?