Avoid Common Prometheus Pitfalls: Best Practices for Reliable Monitoring

Balancing Accuracy and Reliability

Prometheus trades some metric accuracy for simplicity and reliability; short‑term spikes may be missed and values like QPS, P95, P99 are estimates. Users should understand this trade‑off early and communicate it.

Self‑Monitoring Prometheus

Deploy at least two independent Prometheus instances that scrape each other’s metrics. Also ensure Alertmanager is highly available or use a “dead‑man’s switch” alert that always fires to detect alert‑chain failures.

Avoid NFS for Storage

Prometheus does not support NFS; using it can corrupt data and cause loss of historical metrics.

Eliminate High‑Cardinality Metrics Early

High‑cardinality labels consume >50% storage and >80% CPU. Identify them with an alert such as count by (__name__)({__name__=~".+"}) > 10000 and drop offending labels via metric_relabel_configs.

Rate Function and Recording Rule Pitfalls

Applying rate() after a sum() or other aggregation can produce spikes because the result is no longer a pure counter. Record the final value directly in a Recording Rule instead of creating an intermediate metric.

Alert vs. Graph Mismatch

Alert evaluation intervals differ from graph sampling intervals, so alerts may fire while graphs appear normal, or vice‑versa. Reduce graph sampling interval or create a Recording Rule for complex alerts to improve visibility.

group_interval Affects Resolved Notifications

Alertmanager’s group_interval applies to both firing and resolved alerts, causing resolved notifications to be delayed. Adjusting the interval or patching the source can change this behavior.

Final Reminder

Monitoring’s purpose is to keep the business stable, not to chase metric or alert count. Developers should adopt an SRE mindset, focusing on reliability and cost‑effective observability.