Master Zabbix: From Installation to Advanced Custom Monitoring and Alerts

1. Monitoring Overview

1.1 Why Monitor

Alert us in advance when the server has problems.

After a problem occurs, we can find the root cause.

Website/server availability.

1.1.1 Website Availability

In high‑availability (HA) systems, reliability is measured by the number of 9s (3‑5 nines).

n nines represent the proportion of uptime in a year; the following calculations illustrate the differences.

1‑nine (90%): 365 × 0.1 = 36.5 days of possible downtime per year. 2‑nine (99%): 365 × 0.01 = 3.65 days. 3‑nine (99.9%): 365 × 0.001 = 8.76 hours. 4‑nine (99.99%): 365 × 0.0001 = 52.6 minutes. 5‑nine (99.999%): 365 × 0.00001 = 5.26 minutes. 6‑nine (99.9999%): 365 × 0.000001 = 31 seconds.

1.2 What to Monitor

Monitor everything that can be monitored with commands.

1.2.1 Monitoring Scope

1.3.1 Remote Management Cards

If you need remote management, use cards such as Dell iDRAC, HP iLO, IBM IMM.

1.3.2 Hardware Monitoring

2. Installing Zabbix

2.1 Environment Check

2.2.1 Installation Methods

Compile installation (many services, complex environment)

YUM installation (clean environment)

YUM requires a repository, e.g., http://www.cnblogs.com/clsn/p/7866643.html

2.2.2 Server Quick‑Install Script

2.2.3 Client Quick‑Deploy Script

2.3 Connectivity Test

2.3.1 Install zabbix‑get on Server

yum install zabbix-get

3. Web Interface Operations

3.1 Zabbix Web Installation

3.1.1 Access via Browser

http://10.0.0.61/zabbix/setup.php

During detection, view specific error messages for troubleshooting.

Select MySQL database and enter password.

Host and port do not need modification; name is custom.

Confirm information and click Next.

Installation complete, click Finish.

Login page: username Admin, password zabbix (capital A).

3.2 Adding Monitoring Items

3.2.1 Modify Zabbix Server Host

Configuration >> Hosts

Host name must match the actual hostname; visible name is for UI display.

Enable the host after modification.

Host now appears in the management list.

3.2.2 Add New Host

Configuration >> Hosts >> Create Host

Check the enable box.

Add a template, select Linux OS, add small then large.

Two monitoring hosts are now listed.

3.2.3 View Monitoring Data

Monitoring >> Latest Data

Search by IP or name.

All monitoring items are listed.

3.2.4 View Graphs

Monitoring >> Graphs, select host and graph.

4. Custom Monitoring and Alerts

4.1 Custom Monitoring

4.1.1 Description

Zabbix provides the Template OS Linux (Template App Zabbix Agent) with CPU, memory, disk, NIC monitoring. Linking a new host to this template automatically adds these items.

Requirement: limit server login users to three; trigger an alarm when exceeded.

4.1.2 Prerequisite Knowledge

4.2 Implementing Custom Monitoring

4.2.1 Custom Syntax

4.2.2 Agent Registration

4.2.3 Server‑Side Registration (Web)

1. Create Template – Configuration >> Templates >> Create Template

Click Add to create the template.

2. Create Application Set (folder) – Configuration >> Applications >> Create Application.

3. Create Item – Configuration >> Items >> Create Item.

Key is the previously created login‑user.

When creating the item, select the previously created application set.

4.2.4 Create Trigger

Triggers fire when an item reaches a defined condition.

Define expression by selecting the monitoring item; T value is the latest value.

Trigger is now visible.

4.2.5 Create Graph

Graph >> Create Graph – name it and link the items.

4.2.6 Link Template to Host

Configuration >> Hosts – a host can link multiple templates.

4.2.7 View Graphs

4.3 Monitoring Alerts

4.3.1 Third‑Party Alert Platform

http://www.onealert.com – OneAlert provides SMS, WeChat, QQ, phone alerts with scheduling.

4.3.2 OneAlert Configuration

Add an application, select Zabbix.

WeChat alerts require following the public account.

4.3.3 Install OneAlert Agent

4.3.4 Delete OneAlert Agent

1. Delete script from Media Types.

2. Delete created user.

3. Delete user group.

4. Delete action.

4.3.5 Trigger Response – Send Alert

Alerts appear in WeChat and email.

Note: Emails are sent only when status changes (good→bad or bad→good).

4.4 Monitoring Visualization

4.4.1 Aggregated Graphs

Latest Data >> Graphs – customize name and add graphs.

4.4.2 Slideshows

Add slideshow – Monitoring >> Composite Graphs >> Slideshow.

4.5 Template Sharing

4.5.1 Host Sharing

In host page, select all and export.

Import on another server.

4.5.2 Template Sharing

GitHub repository: https://github.com/zhangyao8/zabbix-community-repos

5. Monitoring the Entire Server Fleet

5.1 Requirement

Company has 100 servers; need to monitor all with Zabbix.

5.2 Planning

Standard monitoring: CPU, memory, disk, NIC. Challenge: quickly add 100 machines.

Method 1: Clone hosts.

Method 2: Auto‑registration and auto‑discovery.

Method 3: Use Zabbix API (curl, Python).

Develop a custom ops platform compatible with Zabbix.

5.2.1 API Usage (curl)

5.3 Implementation Details

5.3.1 Hardware, System, Network Monitoring

Monitor all cluster nodes, switches, routers (port traffic or SNMP).

5.3.2 Application Service Monitoring

Backup server – monitor rsync port 873 or simulate file transfer.

NFS server – monitor NFS process or rpc port 111.

MySQL – monitor port 3306 or use Zabbix MySQL template.

Web servers – monitor port 80 or HTTP status.

URL monitoring – use Zabbix web check.

Reverse proxy, PPTP, NTP – monitor respective ports.

Monitor Nginx 7 connection states.

5.3.3 Generic Monitoring Methods

Monitor ports with netstat/ss/lsof.

Monitor processes with ps.

Simulate client usage (curl, SQL queries, memcache set/get).

5.4 Full‑Network Monitoring Deployment

5.4.1 Install Client Script (CentOS 6)

5.4.2 Auto‑Discovery Rules

Add auto‑discovery rule and create discovery action.

5.4.3 Monitor Backup Server

Use net.tcp.listen[port] key and create template.

5.4.4 Monitor NFS Server

Create NFS template using proc.num[…] key.

5.4.5 Monitor MySQL Server

Add MySQL key with credentials.

5.4.6 Monitor Web Server

Create template for nginx service and port 80.

proc.num[<name>,<user>,<state>,<cmdline>] – process count
net.tcp.port[<ip>,port] – TCP connectivity check

5.4.7 Monitor URL

Create a simple check page.

echo ok >> /application/nginx/html/www/check.html

5.4.8 Monitor Reverse Proxy Server

Create custom key.

# cat /etc/zabbix/zabbix_agentd.d/userparameter_nk.conf
UserParameter=keep-ip,ip a |grep 10.0.0.3|wc -l

5.4.9 Monitor Nginx 7 Connection States

Add keys to fetch status from nginx.

UserParameter=nginx_active,curl -s 127.0.0.1/status|awk '/Active/ {print $NF}'
UserParameter=nginx_accepts,curl -s 127.0.0.1/status|awk 'NR==3 {print $1}'
UserParameter=nginx_handled,curl -s 127.0.0.1/status|awk 'NR==3 {print $2}'
UserParameter=nginx_requests,curl -s 127.0.0.1/status|awk 'NR==3 {print $3}'
UserParameter=nginx_reading,curl -s 127.0.0.1/status|awk 'NR==4 {print $2}'
UserParameter=nginx_writing,curl -s 127.0.0.1/status|awk 'NR==4 {print $4}'
UserParameter=nginx_waiting,curl -s 127.0.0.1/status|awk 'NR==4 {print $6}'
EOF

6. Auto‑Discovery and Auto‑Registration

6.1 Auto‑Discovery

Server actively discovers clients; agents can also actively report.

6.1.1 Passive Mode

After installing server and agents (Server=172.16.1.61), configure Auto‑Discovery in the web UI (Configuration >> Auto‑Discovery >> Local network).

6.1.2 Create Discovery Action

Configure action to add and enable discovered hosts.

7. Distributed Monitoring and SNMP

7.1 Distributed Monitoring

Use Zabbix proxies to offload load and support multi‑datacenter monitoring.

zabbix Server → zabbix proxy → agents (LAN 1)
zabbix Server → zabbix proxy → agents (LAN 2)

7.1.2 Proxy Setup

Install proxy, configure MySQL database, set DBPassword, Server, Hostname, and start service.

rpm -ivh http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm
yum install zabbix-proxy-mysql -y
yum install mariadb-server -y
systemctl start mariadb.service
create database zabbix_proxy character set utf8 collate utf8_bin;
grant all privileges on zabbix_proxy.* to zabbix@'localhost' identified by 'zabbix';
zcat /usr/share/doc/zabbix-proxy-mysql-3.0.13/schema.sql.gz | mysql -uzabbix -pzabbix zabbix_proxy
sed -i '162a DBPassword=zabbix' /etc/zabbix/zabbix_proxy.conf
sed -i 's#Server=127.0.0.1#Server=172.16.1.61#' /etc/zabbix/zabbix_proxy.conf
sed -i 's#Hostname=Zabbix proxy#Hostname=cache01#' /etc/zabbix/zabbix_proxy.conf
systemctl restart zabbix-proxy.service

Configure agents to point to the proxy (Server=172.16.1.21).

# grep ^Server /etc/zabbix/zabbix_agentd.conf
Server=172.16.1.21
ServerActive=172.16.1.21
sed -i 's#172.16.1.61#172.16.1.21#g' /etc/zabbix/zabbix_agentd.conf
systemctl restart zabbix-agent.service

7.2 SNMP Monitoring

Install net‑snmp utilities and configure snmpd.

yum -y install net-snmp net-snmp-utils
sed -i '57a view systemview   included  .1' /etc/snmp/snmpd.conf
systemctl start snmpd.service

Test with snmpwalk:

snmpwalk -v 2c -c public 127.0.0.1 sysName
# SNMPv2-MIB::sysName.0 = STRING: m01

Add SNMP host in Zabbix UI, select SNMP template, and view monitored data.

Source: "惨绿少年" (copyright belongs to original author).