Why Prometheus Metrics Have High Cardinality and How to Fix It

Prometheus is a core monitoring component, but as service scale grows the number of unique label combinations (cardinality) can explode, leading to excessive memory usage, slow queries, scrape delays, storage bloat, OOM crashes and unstable services.

What is cardinality?

Cardinality = number of time series = metric name × number of label value combinations. A label with many unique values (e.g., user_id, request_id, timestamps) causes exponential growth. For example, 10 methods × 100 paths × 10 status values = 10,000 series; with a user ID in the path, 1 million users become 100 million series.

Impact on resources

Each series consumes 3‑5 KB memory (1 M series ≈ 3‑5 GB, 10 M ≈ 30‑50 GB, 100 M ≈ 300‑500 GB).

Storage per series is ~1‑2 KB per hour; 1 M series for 15 days ≈ 540 GB.

Query time grows with series count: 1 M series < 1 s, 10 M series 10‑30 s, 100 M series timeout/OOM.

Typical symptoms

Prometheus memory > tens of GB.

Scrape latency increasing, timeouts.

Query latency > seconds.

Rapid growth of storage usage.

Overall troubleshooting flow

发现 Prometheus 性能问题
    ↓
确认是否为基数问题
    ├── 检查时间序列总数
    ├── 检查内存占用趋势
    └── 检查查询延迟
    ↓
定位高基数指标
    ├── 查询 TSDB 状态
    ├── 按基数排序 Top 20
    └── 分析每个指标的标签
    ↓
分析根因 (标签设计、错误指标、动态标签、采集配置)
    ↓
采取处理措施 (删除、重构、relabel、降采集频率、远程存储等)
    ↓
预防措施 (指标规范、代码审查、监控基数趋势)

Step‑by‑step guide

1. Confirm cardinality problem

Check memory: kubectl top pod -n monitoring prometheus-server-xxxxx or ps aux | grep prometheus.

Check series count: query prometheus_tsdb_symbol_table_size_bytes in the UI.

If series > 1 M, likely a cardinality issue.

2. Identify high‑cardinality metrics

topk(20, count by (__name__)({__name__=~".+"}))

Result shows the 20 metrics with the most series.

3. Find high‑cardinality labels

count by (user_id)(http_requests_total)

If the result lists thousands of distinct values, the label is high‑cardinality.

4. Trace metric source

Inspect scrape_configs in prometheus.yml for the job exposing the metric.

Search application code for the metric definition (e.g.,

Counter('http_requests_total', ... ['method','path','status','user_id'])

).

5. Root‑cause analysis

Label contains unique identifiers (user_id, request_id).

Label contains dynamic values (full URL, IP, timestamp).

Label not normalized (e.g., /api/users/123456 instead of /api/users/:id).

Misconfigured relabel_configs that fail to drop high‑cardinality labels.

6. Mitigation methods

Delete the metric if it provides no value:

curl -X POST -g 'http://prometheus:9090/api/v1/admin/tsdb/delete_series?match[]={__name__="http_requests_total"}'

.

Drop high‑cardinality label via relabeling:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'http_requests_total'
    action: drop
  - regex: 'user_id|request_id|session_id'
    action: labeldrop

Aggregate label values (e.g., normalize paths):

# Python example
import re
def normalize_path(path):
    return re.sub(r'/\d+', '/:id', path)

Or in Prometheus relabel:

metric_relabel_configs:
  - source_labels: [path]
    regex: '/api/users/[0-9]+'
    replacement: '/api/users/:id'
    target_label: path

Use exemplars to attach trace IDs without creating extra series:

request_counter.labels(method=method, status=status).inc(exemplar={'trace_id': trace_id})

Split metric into low‑cardinality ones and log high‑cardinality details elsewhere.

Reduce scrape interval for slowly changing metrics (e.g., scrape_interval: 5m).

Shorten retention time (e.g., --storage.tsdb.retention.time=7d).

Remote storage (Thanos, VictoriaMetrics, Cortex) for long‑term high‑cardinality data.

7. Clean up existing data

Delete series via the admin API (as shown above).

Run

curl -X POST http://prometheus:9090/api/v1/admin/tsdb/clean_tombstones

to reclaim space.

Optionally rebuild the TSDB: stop Prometheus, delete /data/prometheus, then restart.

8. Validate the fix

Check series count before and after (e.g., 12,345,678 → 1,234,567).

Verify memory drop (48 GB → 6 GB).

Measure query latency improvement (25 s → 0.8 s).

Confirm storage reduction (540 GB → 60 GB).

Risk warnings

Deleting useful metrics can break alerts; always confirm with stakeholders.

Incorrect relabel_configs regexes may drop needed data.

Data cleanup is irreversible; take TSDB snapshots or remote backups first.

Removing labels changes query results; update dashboards and alerts accordingly.

Maintenance (restart, rebuild) can cause temporary monitoring gaps; use HA deployment and schedule during low‑traffic windows.

Preventive measures

Define a metric naming and label convention (low‑cardinality labels only).

Enforce checks in CI/CD (lint scripts that reject high‑cardinality labels).

Regularly monitor cardinality trends with Grafana dashboards (e.g., prometheus_tsdb_symbol_table_size_bytes, top‑20 metrics, memory usage).

Set alerts for rapid cardinality growth:

- alert: CardinalityIncreasing
  expr: deriv(prometheus_tsdb_symbol_table_size_bytes[1h]) > 10000
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Cardinality increasing rapidly"

Run periodic audit scripts (snapshot, top‑20 report, storage usage) and keep remote backups.

Educate developers on proper label design and use of exemplars.

Best‑practice checklist

Use only low‑cardinality labels (method, status, endpoint).

Never use identifiers (user_id, request_id, IP, full URL) as labels.

Normalize dynamic parts of paths before exposing them.

Prefer exemplars for request‑level tracing.

Separate high‑cardinality data to logs/APM, keep aggregated metrics in Prometheus.