Master Prometheus Metrics: Best Practices for Effective Monitoring

In this article we introduce how to use Prometheus to monitor applications and share practical metrics practices based on experience and official documentation.

Determine Monitoring Targets

Before designing metrics, clarify what needs to be measured according to the problem context, requirements, and the system itself.

Four golden metrics from Google for distributed monitoring are:

Latency: request response time.

Traffic: volume to assess capacity needs.

Errors: count of error requests to gauge error rate.

Saturation: resource constraints affecting service state, e.g., memory usage.

These metrics satisfy four monitoring needs:

Reflect user experience and core performance.

Show system throughput.

Help discover and locate faults.

Indicate system saturation and load.

Choose Vectors

Select vectors when data types are similar but resource types or collection locations differ, and ensure uniform units within a vector. Examples include request latency for different resources, regional server latency, or error counts per HTTP request.

Determine Labels

Common label dimensions include resource, region, type, etc. Labels must be additive and comparable; avoid mixing incompatible units.

my_metric{label=a} 1 my_metric{label=b} 6 my_metric{label=total} 7

Instead, aggregate totals with PromQL on the server side or use a separate metric for totals.

Name Metrics and Labels

Good names are descriptive. Metric names should follow the pattern a-zA-Z*:*, include a domain prefix (e.g., prometheus_notifications_total), and end with a unit suffix (e.g., http_request_duration_seconds, node_memory_usage_bytes). Use base units like seconds or bytes.

Label names reflect dimensions, such as region (shenzhen/guangzhou/beijing), owner (user1/user2), or stage (extract/transform/load).

Select Buckets

Appropriate buckets improve histogram percentile accuracy. Ideally, bucket counts are roughly equal across intervals. Use default buckets ({0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,2.5,5,10}) or exponential buckets for latency with long tails, adjusting based on observed data.

Grafana Tips

View all dimensions : Query only the metric name without calculations and leave Legend format empty to see raw metric data.

Scale linking : In Settings, change Graph Tooltip to "Shared crosshair" or "Shared Tooltip" to link scales across panels, aiding correlation analysis.