How to Build a Systematic Enterprise Monitoring Architecture

Monitoring Technology System

A complete monitoring system can be abstracted into four indispensable modules: Collection + Data + Operators + Alerts .

1. Collection

Collection methods are divided into Agent‑based (plugins, scripts, host logs, processes, APM probes, SDKs) and Agent‑less (SNMP, IPMI/Redfish, SSH, JMX, ODBC/JDBC, Syslog, ICMP, HTTP(s), TCP/UDP, SMTP, etc.).

Collection frequency typically ranges from seconds to minutes; minute‑level is common. While faster collection can detect issues sooner, second‑level sampling increases load on target machines, storage, and computation, potentially outweighing its benefits. The article advises evaluating whether second‑level frequency truly adds business value.

Collection transport can be Pull (active) or Push (passive), and transmitted via direct connection or Proxy. Proxy transport helps solve cross‑network data transfer and alleviates bottlenecks caused by many monitoring nodes.

2. Data

Data types include Metrics, Logs, and Traces. Metrics are numeric, identified by dimensions; Logs are textual and searched for keywords; Traces follow a request’s path to expose latency.

Typical systems: Metrics – Zabbix, Prometheus; Logs – ELK, Splunk; Traces – Dynatrace, SkyWalking.

Data storage options:

Relational databases (MySQL, MSSQL, DB2) – limited for massive monitoring.

Time‑series databases (InfluxDB, OpenTSDB, Prometheus) – designed for metric storage and computation.

Full‑text search databases (Elasticsearch) – suited for log and trace retrieval.

Data views present monitoring data in human‑readable formats for different roles (leadership, administrators, on‑call staff). Common view types: dashboards, topology maps, custom dashboards, reports, and searchable tables.

3. Operators

Data processing includes cleaning (e.g., extracting useful fields from unstructured logs), calculation (e.g., deriving disk usage rate from total and used space), enrichment (adding tags such as host or data‑center), and metric derivation (creating new metrics from existing ones).

4. Alerts

Alert convergence can be achieved through suppression (prevent duplicate alerts), masking (ignore known events like maintenance windows), and aggregation (merge similar alerts, e.g., CPU, memory, and network spikes during a traffic surge).

Alert notification can be sent to people (WeChat, SMS, email) or systems (API push), with extensibility for custom channels.

Monitoring Metric System

The metric system provides a dictionary of indicators that must be measurable, collectible, understandable, and consumable. Indicators are organized into six layers from infrastructure to business operations:

Infrastructure layer – power, environment, security devices.

Hardware layer – servers, storage, network, security equipment.

Operating‑system layer – OS metrics, virtualization platforms, container clusters.

Component‑service layer – databases (MySQL, Oracle, MongoDB, Redis, InfluxDB) and middleware (Tomcat, Kafka, RabbitMQ).

Application‑performance layer – service‑side trace metrics and client‑side web/app metrics.

Business‑operation layer – business‑critical KPIs such as transaction volume, latency, inventory, active users.

Indicators are classified into three grades: core (life‑or‑death), key (alert‑triggering), and regular (analysis). Core indicators should have alarm baselines; key indicators are recommended; regular indicators depend on business scenarios.

Health Management System

Effective monitoring requires a closed‑loop alert management process divided into pre‑alert, in‑alert, and post‑alert stages.

1. Alert Prevention (Pre‑alert)

Activities include performance testing, capacity planning, proactive monitoring setup, and preparing runbooks for disaster recovery, throttling, or degradation.

2. Alert Handling (In‑alert)

The handling workflow consists of alert perception, response,定位 (diagnosis), and recovery. Alerts are first classified into fatal, warning, and reminder levels based on object importance, metric criticality, environment, and business impact.

Perception can be automated (system detection, synthetic probing) or manual (operator observation). Response involves notifying responsible personnel via multiple channels and escalating as needed.

定位 relies on rapid analysis of metrics, logs, and traces, as well as change history, to pinpoint the root cause. Recovery employs isolation, rate‑limiting, service degradation, scaling, or restart actions.

3. Post‑incident Review (Post‑alert)

Post‑mortem includes problem review, knowledge‑base enrichment, and optimization. The article stresses building a searchable knowledge base driven by consumption, integrating tools, and assigning dedicated knowledge‑management teams.

Additional analyses such as alert‑close statistics, false‑alert tagging, alert ranking, and self‑healing effectiveness help refine alert policies.

Operational Management Metrics

Key metrics to evaluate monitoring effectiveness:

Monitoring coverage – ratio of monitored objects and metrics to total inventory.

Alert handling metrics – MTTI (mean time to identify), MTTA (mean time to acknowledge), MTTR (mean time to resolve), MTBF (mean time between failures).

The goal is to reduce MTTA and MTTR while increasing MTBF, thereby improving fault detection speed, resolution speed, and overall business continuity.

By following the above systematic approach, enterprises can achieve comprehensive, sustainable monitoring that supports proactive operations and continuous improvement.