Why InfluxDB’s max‑value‑per‑tag Error Occurs and How to Resolve It
This article explains the cause of InfluxDB’s max‑value‑per‑tag error when Prometheus remote‑writes high‑cardinality tags, analyzes why the built‑in memory index triggers OOM protection, and presents three practical solutions—including moving indexes to disk, storing high‑cardinality tags as fields, and filtering them before write—to ensure stable monitoring data persistence.
Problem Scenario
InfluxDB is widely used as a time‑series database for persisting monitoring data. During write operations, the max‑value‑per‑tag error can appear, which is a protective measure to prevent out‑of‑memory (OOM) crashes when the number of distinct tag values grows too large.
The article examines this error from the perspective of Prometheus remote‑write to InfluxDB, illustrating the error with screenshots and describing the impact on continuous monitoring.
Root Cause Analysis
When Prometheus writes metrics to InfluxDB, tags with high cardinality—such as a globally unique trace ID or a transaction timestamp—are included. These tags cause the following chain of events:
Exporter exposes metrics that contain high‑cardinality tag elements.
Prometheus continuously streams these metrics to InfluxDB, using Snappy compression and protobuf for transport.
InfluxDB parses the incoming stream, treating the measurement name as the table name, tags as index fields, and field values as regular columns.
All tag values are stored in memory by default to speed up index lookups. To avoid excessive memory consumption, InfluxDB enforces the max‑values‑per‑tag limit (default 100 000). When a tag exceeds this threshold, write operations fail with the protective error. Consequently, after a period of continuous writes, the number of distinct high‑cardinality tag values reaches the limit, causing subsequent writes to be rejected.
Solutions
Solution 1 – Move Indexes to Disk
InfluxDB allows configuring the index storage engine to persist indexes on disk instead of keeping them entirely in memory. By setting index-version=tsi1 in the configuration file, the database creates an index directory under /var/lib/influxdb/data . This eliminates the OOM protection trigger but increases disk usage. Testing under a simulated 100 TPS workload showed that after 12 hours the system remained stable, while disk consumption grew by roughly 300 MB per hour (index ≈ 50 MB, TSM ≈ 120 MB, _series ≈ 130 MB).
Solution 2 – Store High‑Cardinality Tags as Fields
Since high‑cardinality tags are mainly used for alert context rather than analytical dimensions, they can be demoted to regular fields. This requires customizing InfluxDB’s write path: when a remote‑write request arrives, retain only the measurement name as the index and convert the high‑cardinality tags into fields. After applying this change and running the same 12‑hour test, the max‑value‑per‑tag warning disappeared and disk growth was limited to a few tens of megabytes. However, because multiple metrics share the same batch timestamp, InfluxDB treated them as updates to the same series, resulting in data loss for the monitoring use‑case.
Solution 3 – Filter High‑Cardinality Tags Before Write
The final approach is to prevent high‑cardinality tags from reaching InfluxDB at all. By editing Prometheus’s remote_write configuration, tags identified as high‑cardinality are stripped from the payload before it is sent. Running the same 12‑hour workload confirmed that InfluxDB operated normally, without the max‑value‑per‑tag warning, and resource consumption remained moderate.
Conclusion
The root cause of the max‑value‑per‑tag error is the presence of high‑cardinality tags in monitoring metrics, which InfluxDB is not designed to index efficiently. For stable operation, such tags should either be stored on disk (accepting higher storage costs), demoted to fields (risking update‑collision issues), or filtered out before ingestion. In practice, filtering high‑cardinality tags offers the most balanced solution for monitoring data persistence.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
dbaplus Community
Enterprise-level professional community for Database, BigData, and AIOps. Daily original articles, weekly online tech talks, monthly offline salons, and quarterly XCOPS&DAMS conferences—delivered by industry experts.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
