Why Metric Counts Explode from Thousands to Millions in High‑QPS Systems

When monitoring was simple

In a monolithic setup a few core metrics—QPS, P99 latency, error rate, CPU, memory, GC pause—were scraped by a single Prometheus instance from /metrics endpoints and stored locally, easily handling thousands of time series.

What metric count really counts

The number of "metrics" is not the count of metric names; it is the count of active time‑series (cardinality). Each series equals a metric name plus a unique set of label key‑value pairs, e.g.

http_requests_total{service="order", method="GET", status="200", pod="order-7d9-abc"}

versus

http_requests_total{service="order", method="GET", status="500", pod="order-7d9-abc"}

. The cardinality is the Cartesian product of all label value counts.

Linear growth vs. product explosion

Micro‑service scaling causes a linear increase in series (more services × metrics × pods). This can be handled with sharding or federation. However, adding an unbounded high‑cardinality label such as user_id multiplies the series count dramatically—one label can add millions of series instantly, as illustrated by the OOM crash after adding user_id to an HTTP metric.

Five directions of explosion

When active series reach millions, five pressure points appear simultaneously:

Storage explosion: Prometheus keeps head series in memory; index size grows super‑linearly, leading to OOM.

Query overload: High‑cardinality aggregations force the PromQL engine to scan massive series, causing slow or OOM queries.

Scrape pressure: A single Prometheus cannot fetch all series within its scrape interval; remote‑write queues back up.

Cost runaway: Self‑hosted TSDBs consume CPU, memory, and disk; SaaS providers charge per custom metric or host, inflating bills.

Signal‑to‑noise collapse: Only a few dozen series are truly useful; the rest generate alert storms and obscure real issues, increasing MTTR.

Governance: control cardinality first

Adding hardware only treats the symptom. The fundamental fix is to reduce cardinality at the source:

Eliminate high‑cardinality labels ( user_id, request_id, full URLs, etc.). Use tracing and logging for those details.

Template variable parts of URLs (e.g., /user/:id) instead of raw IDs.

Set a cardinality budget per metric and drop or relabel excess series at scrape time.

Adopt RED/USE golden‑metric frameworks and instrument only the most valuable signals.

A practical checklist distinguishes "low‑cardinality, enumerable" dimensions (method, status, region) from unbounded ones.

Distribute the TSDB

After cardinality is tamed, normal growth still exceeds a single node. Horizontal‑scalable back‑ends such as Thanos, Cortex, Grafana Mimir, VictoriaMetrics, or Uber’s M3DB store data in object storage (S3) and provide unlimited capacity.

Key techniques include:

Horizontal backend: remote‑write from Prometheus to a scalable store.

Downsampling + tiered retention: Keep high‑resolution data for recent days, low‑resolution for long‑term.

Recording rules (pre‑aggregation): Materialise low‑cardinality aggregates for dashboards and alerts.

Collector‑side aggregation: Use OpenTelemetry Collector or sidecars to drop or aggregate high‑cardinality series before they reach storage.

Don’t use metrics as a detail database

Observability’s three pillars have distinct responsibilities:

Metrics: Low‑cardinality aggregates for health, capacity, and alerts.

Traces: High‑cardinality request‑level paths, sampled to control cost.

Logs: Structured event details searchable by trace_id or user_id.

Embedding raw IDs in metric labels violates this separation and triggers cardinality explosions.

Observability FinOps

Monitoring itself incurs storage, query, and human‑operational costs. Effective governance requires:

Measuring each metric’s cardinality and query frequency.

Removing unused (zombie) metrics.

Dropping or aggregating high‑cardinality metrics at collection time.

Metrics should prove their value; otherwise they become costly noise.

Takeaway

Metric count growth from thousands to millions is driven by multiplicative cardinality, not by merely adding more points. The root cause is unbounded high‑cardinality labels. Controlling cardinality, moving to a distributed TSDB, and assigning detailed data to traces and logs resolves the five‑fold pressure and restores observability effectiveness.