What Does Ops Monitoring Cover? Key Metrics, Golden Signals & Implementation

Monitoring Objects: Four Core Layers

The core goal of ops monitoring is to ensure business system stability, availability, and performance. To achieve this, monitoring must cover four layers:

Infrastructure Monitoring : Physical/virtual servers (CPU, memory, disk I/O, disk space, network traffic, load average), network devices (port status, bandwidth utilization, packet loss/errors, CPU load), storage devices (disk health, IOPS, throughput, latency).

Application Monitoring : Service status (web servers like Nginx/Apache, app servers like Tomcat/JVM, databases like MySQL/Redis), application performance (API response time, throughput QPS/TPS, error rate, key business logic latency), code-level performance (method execution time, SQL query efficiency, slow transaction tracing).

Middleware & Service Monitoring : Databases (connection count, query cache hit rate, slow queries, lock waits, master-slave sync), message queues (queue length, produce/consume rates, consumer lag), caches (hit rate, memory usage, network bandwidth, key count).

Business-Level Monitoring : Core business metrics (DAU, order volume, revenue, payment success rate, registration conversion), business funnel models (conversion rates from click to purchase), user experience monitoring (page load time, frontend errors, geo network latency).

Key Metrics: The Four Golden Signals

Google SRE's universally accepted "Four Golden Signals" apply to most monitoring scenarios:

Latency : Time to process a request. Distinguish successful vs failed requests — a failed request returning immediately can skew averages. Examples: API average response time, P99 latency (slowest 1%).

Traffic : System load. Examples: HTTP requests per second (QPS), database transactions per second (TPS), network throughput (MB/s).

Errors : Request failure rate. Examples: HTTP 5xx/4xx counts, application log exceptions, service call timeouts.

Saturation : Resource utilization level, often a precursor to bottlenecks. Examples: CPU, memory, disk usage, queue length (e.g., message queue backlog).

Additional Important Metrics : Availability (service accessibility, measured in "nines" like 99.99%), performance counters (JVM GC frequency/duration, database connection pool active connections).

How to Monitor Applications: Process & Tools

1. Define Monitoring Objectives

Why : Stability assurance, performance bottleneck detection, or user experience optimization.

What : Select specific metrics based on goals (reference golden signals). Prioritize core business paths and APIs.

2. Choose Monitoring Stack (Multi-Tool Combination)

Metrics Monitoring : Collect/store time-series data. Representative: Prometheus (mainstream, open-source), Zabbix, Open-Falcon. Workflow: Application exposes metrics → Prometheus scrapes/stores → Grafana visualizes → Alertmanager alerts.

Log Monitoring : Collect, store, retrieve application logs. Representative: ELK/EFK Stack (Elasticsearch, Logstash/Filebeat, Kibana), Loki (lightweight, integrates with Grafana).

Distributed Tracing : Track request paths across services for latency analysis. Representative: Jaeger , SkyWalking, Zipkin.

User Experience Monitoring : Real-user frontend experience. Representative: Google Analytics, FrontJS, Tingyun.

3. Implementation Example: Spring Boot + Prometheus

Step 1: Application Instrumentation & Metrics Exposure

Add Micrometer dependency — Java's metrics facade for easy Prometheus integration.

Micrometer provides auto-configured metrics for JVM (memory, threads, GC), database connection pools (HikariCP), HTTP endpoints (MVC).

Application exposes /actuator/prometheus endpoint for Prometheus scraping.

Step 2: Deploy & Configure Prometheus

Deploy Prometheus server.

In prometheus.yml, add application as scrape target:

scrape_configs:
  - job_name: 'springboot-app'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['your-app-host:8080']

Step 3: Visualization & Alerting

Deploy Grafana , add Prometheus as data source.

Build dashboards visualizing key metrics (QPS, latency, error rate, JVM memory).

Configure alert rules in Grafana or Prometheus Alertmanager, e.g.:

When "error rate > 5% for 1 minute" → trigger PagerDuty/DingTalk/SMS alert.

When "API P99 latency > 2s for 2 minutes" → trigger alert.

4. Alert Handling & Continuous Improvement

Alert Tiering : Classify by urgency (P0-critical, P1-important, P2-warning) to avoid alert fatigue.

Runbooks : Write handling guides for each alert to direct on-call engineers through initial triage.

Closed-Loop Management : Post-incident reviews to improve monitoring (earlier detection? missing metrics?) and continuously refine the system.

Summary Reference

Infrastructure : CPU, memory, disk, network — Tools: Node Exporter, Zabbix Agent

Application : API latency, error rate, QPS, JVM GC — Tools: Micrometer, Prometheus, JMX

Logs : Error logs, exception stacks, business logs — Tools: ELK/EFK, Loki

Tracing : Cross-service request latency, call relationships — Tools: SkyWalking, Jaeger

Business : Order volume, payment success rate, DAU — Tools: Custom instrumentation, big data platforms

An excellent monitoring system is not a mere pile of tools, but an organic whole that is business-centric, metric-driven, full-chain covering, and enables rapid problem localization . It requires collaboration across ops, development, and business teams to build and maintain.