Mastering Application Monitoring with Prometheus: Practical Tips and Best Practices

Mastering Application Monitoring with Prometheus

In this article we introduce how to use Prometheus to monitor applications, summarizing practical metrics based on our experience and the official documentation.

Identify What to Monitor

Before designing metrics, clearly define the objects to be measured according to the problem background, requirements, and the system itself.

Four Golden Metrics

Latency : time taken to serve a request.

Traffic : volume of traffic to gauge service capacity.

Errors : rate of error requests.

Saturation : degree to which a constrained resource (e.g., memory) limits the service.

These metrics satisfy four monitoring needs:

Reflect user experience and core performance (e.g., request latency, job completion time).

Show system throughput (e.g., request count, network packet size).

Help discover and locate faults (e.g., error count, failure rate).

Indicate system saturation and load (e.g., memory usage, queue length).

Additional custom metrics can be added for specific scenarios, such as measuring the latency and failure count of a frequently called library interface.

Classify the Application Type

Online‑serving systems: require immediate responses (e.g., web servers).

Offline processing systems: jobs run without waiting for a response (e.g., Spark batch jobs).

Batch jobs: one‑off tasks that finish after execution (e.g., MapReduce analysis).

Each type has typical measurement objects, for example online services focus on request count, error count, and latency.

Select Vectors

Choose vectors (resource dimensions) that share the same data type but differ in source or collection point, and keep the unit consistent.

Examples include measuring request latency across different resources, regions, or HTTP error counts.

Define Labels

Common label dimensions are resource, region, type, etc. Labels must be additive and comparable; mixing totals with individual dimensions is discouraged.

my_metric{label=a} 1 my_metric{label=b} 6 my_metric{label=total} 7

Instead, aggregate totals on the server side using PromQL or separate metrics.

Name Metrics and Labels

Good names are self‑descriptive. Metrics should follow the pattern [a-zA-Z_]+, include a domain prefix, and end with a unit suffix (e.g., http_request_duration_seconds, node_memory_usage_bytes).

Labels are named after the dimension they represent, such as region=shenzhen, owner=user1, or stage=extract.

Choose Buckets for Histograms

Appropriate buckets improve percentile accuracy. Use default buckets ({0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10}) or exponential buckets, adjusting based on observed data distribution.

Grafana Tips

View All Dimensions

To see every available dimension, query only the metric name without calculations and leave the legend format empty.

Ruler Synchronization

In Grafana Settings, change the Graph Tooltip to Shared crosshair or Shared Tooltip to link rulers across panels, making it easier to correlate two metrics.