Mastering System and Application Monitoring with the USE Method and Prometheus

1. Introduction

A good monitoring system not only exposes real‑time issues but also automatically analyzes and locates bottlenecks, reporting them to the responsible teams. The core of effective monitoring is a set of comprehensive, quantifiable metrics covering both system resources and application behavior.

System‑level monitoring must include overall resource usage such as CPU, memory, disk, file system, and network. Application‑level monitoring must capture process CPU, disk I/O, as well as request latency, errors, and internal object memory usage.

2. System Monitoring

1. USE Method

Before building a monitoring system, you want a concise way to describe resource usage. The USE (Utilization, Saturation, Errors) method simplifies performance metrics into three categories.

Utilization – the percentage of time or capacity a resource is used for service.

Saturation – the degree of resource busy‑ness, often related to queue length.

Errors – the count of error events; more errors indicate more severe problems.

These three categories cover common performance bottlenecks for hardware resources (CPU, memory, disk, network) and software resources (file descriptors, connections, connection tracking).

2. Performance Metrics

The following table (shown in the image) lists typical performance metrics for each resource.

While USE focuses on core bottleneck indicators, other metrics such as system logs, process resource usage, and cache usage remain important for auxiliary analysis.

3. Monitoring System Architecture

A complete monitoring system consists of data collection, storage, query/processing, alerting, and visualization modules. Open‑source tools like Zabbix, Nagios, and Prometheus can be used.

Below is the basic architecture of Prometheus.

Data collection: Prometheus targets are the objects to scrape; Retrieval pulls metrics via HTTP (pull mode) or receives them via Push Gateway (push mode).

Data storage: TSDB (time‑series database) persists metrics on disk, optimized for high‑volume, append‑only writes.

Query and processing: TSDB provides PromQL, a concise query language for filtering, aggregation, and basic processing, serving as the foundation for alerts and dashboards.

Alerting: AlertManager handles alert rules, grouping, inhibition, and silencing to avoid alert fatigue.

Visualization: Prometheus’s web UI offers basic graphs; combined with Grafana it delivers powerful dashboards.

4. Summary of System Monitoring

The core of system monitoring is resource usage (CPU, memory, disk, file system, network, file descriptors, connections, etc.). The USE method reduces metrics to utilization, saturation, and error count, allowing quick identification of performance bottlenecks when any of these values are high.

By integrating these metrics into a full monitoring pipeline—from collection to storage, querying, alerting, and visualization—you can expose bottlenecks, track historical data, and pinpoint root causes.

3. Application Monitoring

1. Application Metrics

Application‑level monitoring focuses on request count, error rate, and response latency—key indicators of user experience and service reliability.

Additional essential metrics include process resource usage (CPU, memory, I/O, network), inter‑service call statistics (frequency, errors, latency), and internal logic performance (critical path timings, error counts).

Collecting these metrics with a system like Prometheus + Grafana enables both alerting and visual analysis of application health.

2. Full‑Chain Tracing

Distributed tracing tools such as Zipkin, Jaeger, and Pinpoint build a full‑chain trace across multiple services, helping locate the exact component causing latency or failures.

Tracing also generates topology maps that are invaluable for analyzing complex micro‑service architectures.

3. Log Monitoring

Metrics alone may miss contextual information; logs provide detailed strings that capture the exact circumstances of events. The classic ELK stack (Elasticsearch, Logstash, Kibana) is used for log collection, indexing, and visualization.

Logstash ingests and preprocesses logs, Elasticsearch indexes them for fast full‑text search, and Kibana visualizes the results. In resource‑constrained environments, Fluentd (EFK stack) can replace Logstash.

4. Summary of Application Monitoring

Application monitoring consists of metric monitoring—measuring performance indicators over time—and log monitoring—providing contextual details via ELK. In complex, multi‑service scenarios, full‑chain tracing adds dynamic call‑graph insights, accelerating root‑cause analysis.