Comprehensive Guide to Monitoring Systems, Tools, and Best Practices

Introduction Monitoring is a critical component of the entire product lifecycle, enabling proactive fault detection and providing detailed data for post‑incident analysis. The article introduces the importance of monitoring and suggests that readers can gain a deep understanding of monitoring architectures.

Monitoring Objectives The primary goals include continuous real‑time system monitoring, real‑time status feedback, ensuring service reliability and safety, and maintaining stable business operation even when failures occur.

Monitoring Methods Key steps are: understanding the monitoring target, defining performance baseline metrics, setting alarm thresholds, and establishing fault‑handling procedures.

Monitoring Core The core workflow consists of problem discovery, problem localization, problem resolution, and post‑mortem summarization.

Monitoring Tools A classification of tools is presented, ranging from classic solutions such as MRTG, Cacti, Nagios, and Smokeping to modern platforms like Zabbix, Open‑Falcon, LEPUS TianTu, and various third‑party services. Each tool’s main features and typical use cases are briefly described.

Monitoring Process (Zabbix‑Centric) The recommended workflow using Zabbix includes data collection (SNMP, Agent, ICMP, SSH, IPMI), data storage (MySQL or other databases), data analysis, data visualization (web UI, mobile apps), alert notification (phone, email, WeChat, SMS), and alert handling based on severity.

Monitoring Metrics Metrics are grouped into hardware, system, application, network, traffic analysis, log, security, API, performance, and business monitoring. Each category lists typical indicators (e.g., CPU usage, memory, disk I/O, network throughput, service status, etc.).

Hardware Monitoring IPMI can monitor CPU, memory, disk, temperature, fan speed, and voltage. Example Zabbix template: Zabbix IPMI Interface.

System Monitoring Focuses on Linux server resources such as CPU load, context switches, memory usage, disk I/O, and network traffic. Zabbix provides the Zabbix Agent Interface template for these metrics.

Application Monitoring Covers services like LVS, HAProxy, Docker, Nginx, PHP‑FPM, Memcached, Redis, MySQL, RabbitMQ, etc. Zabbix offers Zabbix Agent UserParameter for custom checks and Zabbix JMX Interface for Java applications; Percona plugins are mentioned for MySQL.

Network Monitoring Emphasizes the need for network status visibility across data centers, recommending Smokeping for latency and packet loss visualization.

Traffic Analysis Discusses web analytics (Baidu Tongji, Google Analytics, Piwik) for understanding user behavior and marketing effectiveness.

Log Monitoring Suggests using the ELK stack (Elasticsearch, Logstash, Kibana) to collect, store, query, and display system and application logs, with optional Zabbix log‑based alerts.

Security Monitoring Mentions host‑level firewalls (iptables), web WAFs (Nginx+Lua), and third‑party vulnerability services for comprehensive protection.

API Monitoring Recommends tracking API request methods, availability, correctness, and response time as key performance indicators.

Performance Monitoring Covers website performance metrics such as DNS response time, HTTP connection time, page load index, and overall availability. Zabbix’s web monitoring feature ( Zabbix Web Monitoring) is highlighted.

Business Monitoring Stresses the importance of monitoring core business KPIs (order volume, user registrations, active users, promotion impact, etc.) and visualizing them in dashboards.

Alerting Describes common alert channels like SMS and email, with examples of Zabbix‑generated notifications.

Alert Handling Outlines automatic remediation (e.g., restarting a failed Nginx service) and manual escalation based on fault severity and business impact.

Interview Guidance Provides a concise answer framework for interview questions on monitoring, covering hardware, system, service, network, security, web, log, business, traffic analysis, visualization, and automation aspects.

Distributed Monitoring Notes that large enterprises often develop custom monitoring solutions (e.g., Xiaomi’s Open‑Falcon) or combine open‑source components like Sensu, InfluxDB, and Grafana for a tailored platform.

Conclusion While existing open‑source tools are powerful, many organizations build proprietary monitoring stacks to meet specific scalability and feature requirements.