How to Build a Cost‑Effective, Multi‑Layer Monitoring System for Distributed Applications

Why Monitoring Is the Last Line of Defense in Distributed Systems

Monitoring is compared to a health check‑up: without a complete set of examinations, problems can be missed even if individual tests look normal. The article starts with a typical dialogue between an operations person and a developer to illustrate the gap between perceived and actual system performance.

Three Levels of Monitoring

1. Environment Metrics

These include network I/O, latency, disk I/O, disk usage, CPU usage, memory usage, swap usage, etc., which indicate whether the underlying infrastructure is stable. The author suggests choosing one of two simple solutions:

Use Zabbix, a mature enterprise‑grade monitoring product with abundant online installation guides.

If you prefer a more customizable open‑source option, consider open‑falcon (https://github.com/open-falcon/falcon-plus), a Chinese open‑source project with decent activity.

2. Program Metrics

Beyond environment metrics, program metrics cover error counts, request volume, and average response time. Achieving “non‑intrusive” collection is challenging because it often requires instrumenting code. The article recommends leveraging existing unified components such as a gateway, RPC framework, or database access layer to add monitoring hooks, or using AOP to reduce manual instrumentation.

Collected data should be stored in a time‑series database; popular choices are Prometheus (≈23 k stars on GitHub), InfluxDB, or OpenTSDB. Visualization can be done with Grafana. For large‑scale log volumes, use log‑shipping tools like flume or logstash instead of sending logs directly to remote databases.

3. Business Metrics

Business metrics reflect the health of the actual service (e.g., conversion rates, user actions). They are the most valuable but also the most intrusive because they usually require explicit instrumentation (“埋点”). For low‑traffic systems (<1 M PV), pulling data directly from the business database can be a quick workaround; for larger systems, replicate data to a separate read‑only store or use a dedicated monitoring pipeline.

The three layers form a pyramid: business > program > environment in terms of monitoring value, while the cost hierarchy is the opposite.

Practical Evolution Roadmap

The author’s universal advice is to start with environment metrics because they are cheap and easy to implement. Then add business‑level monitoring via direct database queries where feasible, followed by program‑level metrics, and finally complete the “立体化监控” (three‑dimensional monitoring) by filling any gaps.

Alerting Strategy

Effective alerting prevents the system from becoming a “noise generator.” The strategy consists of four key points:

Define clear alert severity levels.

Set alert frequency and implement deduplication/aggregation (convergence).

Choose appropriate notification channels (SMS, mobile push, email, etc.) for each severity.

Specify the recipients and escalation paths (e.g., rotation or hierarchical escalation).

While AI‑enhanced alerting is emerging, the article advises adopting it gradually.

Conclusion

The article summarizes the three‑layer monitoring approach, recommends a smooth progression from environment to program to business metrics, and emphasizes the importance of a well‑designed alerting mechanism to make monitoring actionable.