Mastering Enterprise Monitoring: From Basics to Advanced Toolchains

Introduction

Monitoring is the most important part of operations and the product lifecycle, providing early warnings before incidents and detailed data for post‑mortem analysis.

1. Monitoring Objectives

Continuous real‑time monitoring : keep the system under constant observation.

Real‑time status feedback : instantly see whether a component is normal, abnormal, or failed.

Ensure service reliability and safety : guarantee that systems, services, and business run correctly.

Maintain business continuity : receive alerts immediately when failures occur and resolve them promptly.

2. Monitoring Methods

Identify monitoring objects : know what you are monitoring, e.g., CPU operation.

Define performance metrics : decide which attributes to track, such as CPU usage, load, user‑mode, kernel‑mode, context switches.

Set alarm thresholds : determine when a metric indicates a fault and should trigger an alert.

Fault handling process : establish an efficient workflow for responding to alerts.

3. Core Monitoring Process

Discover the problem : receive a fault alarm.

Locate the problem : analyze alarm details to pinpoint the cause.

Resolve the problem : address the issue according to its priority.

Summarize the problem : document causes and preventive measures.

4. Monitoring Tools

Classic open‑source tools include:

MRTG – network traffic grapher written in Perl, using SNMP for data collection.

Ganglia – scalable distributed monitoring system for clusters.

Cacti – PHP/MySQL based graphing tool built on RRDtool.

Nagios – enterprise‑grade service and host monitoring with alerting.

Smokeping – network latency and packet loss visualizer.

OpenTSDB – time‑series database on HBase for massive metric storage.

Flagship tools:

Zabbix – distributed monitoring platform supporting agents, SNMP, IPMI, JMX, and custom scripts.

Open‑Falcon – open‑source, internet‑grade monitoring system from Xiaomi.

5. Zabbix‑Based Monitoring Workflow

Data collection : Zabbix gathers metrics via SNMP, agents, ICMP, SSH, IPMI, etc.

Data storage : metrics are stored in MySQL (or other databases).

Data analysis : historical data can be visualized and used for root‑cause analysis.

Data presentation : web UI (or mobile apps) displays dashboards.

Alerting : phone, email, WeChat, SMS, and escalation mechanisms.

Alert handling : prioritize and assign incidents based on severity.

6. Monitoring Metrics

6.1 Hardware Monitoring

Use IPMI to monitor power, temperature, fan speed, voltage, and set alarm thresholds for CPU, memory, disks, etc.

6.2 System Monitoring

Key system metrics include CPU usage, load, user‑mode/kernel‑mode ratio, context switches, memory usage and swap, disk I/O, network I/O, and process information. Common tools:

htop

,

top

,

vmstat

,

iostat

,

sar

,

glances

. Zabbix provides templates such as

Zabbix Agent Interface

.

6.3 Application Monitoring

Monitor services like LVS, HAProxy, Docker, Nginx, PHP‑FPM, Memcached, Redis, MySQL, RabbitMQ using Zabbix agents, custom scripts, or dedicated plugins (e.g., percona‑monitoring‑plugins).

6.4 Network Monitoring

Smokeping visualizes latency, packet loss, and round‑trip times across multiple sites.

6.5 Traffic Analysis

Web analytics (Baidu, Google, Piwik) provide visitor, conversion, and region statistics.

6.6 Log Monitoring

ELK stack (Logstash + Elasticsearch + Kibana) collects, stores, searches, and visualizes system and application logs; Zabbix can also filter error logs for alerts.

6.7 Security Monitoring

Combine host‑level firewalls (iptables), web‑level WAF (Nginx + Lua), and third‑party security services; feed alerts into ELK for visualization.

6.8 API Monitoring

Track API endpoints (GET, POST, PUT, DELETE, HEAD, OPTIONS) for availability, correctness, and response time.

6.9 Performance Monitoring

Zabbix Web monitoring (

Zabbix Web 监控

) measures DNS response, HTTP connection time, page load index, and overall availability.

6.10 Business Monitoring

Key business KPIs such as orders per minute, registrations, active users, promotion traffic, and revenue are fed into Zabbix dashboards for real‑time visibility.

7. Alerting Channels

Common channels include SMS, email, phone calls, and instant messaging platforms.

8. Alert Handling

Automatic escalation can restart failed services (e.g., Nginx) while severe incidents are assigned to on‑call engineers based on severity and impact.

9. Interview Preparation

Typical interview questions cover hardware, system, service, network, security, log, traffic, visualization, automation, and business monitoring topics, with suggested answers and best practices.

Conclusion

While many open‑source monitoring solutions exist, large‑scale enterprises often build custom platforms (e.g., Open‑Falcon, Sensu) and combine InfluxDB + Grafana to meet specific requirements.