Zabbix Interview Guide: 8 Questions on Modes, Agent2, Triggers, LLD, Proxy & Prometheus

Interview Self-Test

Q1: Zabbix active vs passive mode differences? Why default to Active in production? (Reference: V. Collection Mechanism)

Q2: Zabbix Agent vs Agent2 differences? Which to choose for new deployment? (Reference: III. Core Architecture)

Q3: Explain Zabbix data model (Host/Item/Trigger/Template) and difference from Prometheus label model (Reference: IV. Data Model)

Q4: Write trigger expression: alert when disk free space <10% and 3-min average <10% (Reference: IV. Data Model)

Q5: Is LLD same as Prometheus Service Discovery? What problems does each solve? (Reference: VII. Templates & LLD)

Q6: What is Zabbix Proxy's role? Does it solve same problem as Server HA cluster? (Reference: III. Core Architecture, X. High Availability)

Q7: How to choose between Zabbix and Prometheus? Common coexistence patterns in enterprises? (Reference: XI. Selection)

Q8: Production SNMP intermittent timeout causing graph gaps: full troubleshooting and solution (Reference: XII. Pitfall Guide)

Reference Answers

Q1: Active vs Passive Mode Differences and Why Production Defaults to Active

Passive Mode:

Server initiates connection to Agent port 10050, sends "get" requests

Agent listens on 10050, responds when asked

Server manages scheduling "when to ask whom", must maintain many inbound connections

Server --TCP--> Agent:10050 --> "get system.cpu.load"
Agent --> "0.42"

Active Mode:

Agent starts, connects to Server port 10051, requests active checks, receives list of Items to collect

Agent collects locally per Item intervals, then batches sender data push to Server

Server no longer actively connects to Agent, no polling scheduler needed

Agent --TCP--> Server:10051 --> "active checks?host=h1"
Server --> "[{key:cpu.load,delay:60s}, ...]"
Agent --> collect then batch push --> Server:10051 "sender data"

Core Differences:

Connection Initiator: Passive: Server initiates; Active: Agent initiates

Port: Passive: Agent 10050; Active: Server 10051

NAT Compatibility: Passive: Poor (Agent behind NAT cannot be reached by Server); Active: Good (outbound connections usually allowed)

Server Scheduling: Passive: Must maintain polling; Active: No polling, passive receive

Latency Stability: Passive: Affected by Server Poller scheduling; Active: Agent pushes at fixed intervals, more stable

Remote Commands: Passive: Immediate execution; Active: Agent learns command only at next batch poll

Why Production Defaults to Active:

NAT/Firewall Friendly: Cloud hosts, dedicated lines, containers — outbound connections usually allowed, inbound often restricted. Active requires no inbound port on Agent.

Stable Latency: Passive alerts vary in timing (peak polling delays). Active pushes at fixed intervals, predictable alert timing.

Server Scalability: Server doesn't maintain polling threads per Agent, passive receive scales horizontally.

Simple Security Model: Monitored host only outbound to Server, inbound ports only for internal network.

Scenarios Where Passive Still Needed:

Debugging with zabbix_get for instant value retrieval

Action remote commands requiring immediate execution (Active waits for next batch poll)

Few critical checks needing Server to fetch instantly

Reference: Zabbix Full Analysis - V. Collection Mechanism

Q2: Zabbix Agent vs Agent2 Differences? Which for New Deployment?

Language: Legacy Agent: C; Agent2: Go

Concurrency: Legacy: in-process multi-thread, single port; Agent2: goroutine concurrency, single process multi-goroutine

Plugins: Legacy: UserParameter scripts; Agent2: built-in plugins + custom plugin interface

Restart Impact: Legacy: all monitoring interrupted; Agent2: fast restart, connections kept alive

Active Mode: Both supported; Agent2 adds disconnect reconnect and batching

Resource Usage: Legacy: low; Agent2: slightly higher but more memory efficient

Logging: Legacy: log files; Agent2: supports log redirection, structured logging

Core Differences:

Language & Concurrency Model: Legacy Agent is C multi-threaded; Agent2 is Go goroutine model. Goroutine context-switch cost far lower than threads, Agent2 more efficient at high Item counts.

Plugin System: Legacy Agent extends via UserParameter — essentially Server calls a shell/script on Agent, high startup overhead, error-prone. Agent2 has native plugin interface (Go), compiled into binary, better performance and stability.

Disconnect Recovery: Agent2 optimizes Active mode with disconnect reconnect and batch push, network jitter results in retransmission not data loss.

Restart Impact: Legacy Agent restart interrupts monitoring for tens of seconds; Agent2 restarts fast, keeps connections alive, shorter interruption.

Why New Deployments Should Choose Agent2:

Goroutine model performs better under high-concurrency collection

Plugin-based extension fits modern ops needs (native plugins for Redis, Docker, Kubernetes, etc.)

Active mode optimizations better suit cloud elastic scenarios

Official focus shifted to Agent2, legacy Agent enters maintenance mode

Only Scenario to Keep Legacy Agent: OS too old (e.g., CentOS 6), libc version incompatible, embedded devices with extreme resource constraints unable to run Go binary.

Reference: Zabbix Full Analysis - III. Core Architecture

Q3: Zabbix Data Model (Host/Item/Trigger/Template) vs Prometheus Label Model

Zabbix Four Core Entities:

Host Group
 └── Host (machine/device/instance)
      ├── Interface (IP+port+protocol)
      └── linked Template
           ├── Item (specific metric)
           │    └── Trigger (anomaly judgment expression)
           └── LLD Rule (auto-discovery rule)
                └── Item Prototype -> real Item

Host: Logical monitored object, core attributes: Host name, Interfaces. A physical machine, switch, DB instance can each be a Host.

Item: Specific monitoring item defining "what to collect, how, how often, value type, retention". Item key globally unique, e.g., system.cpu.load[all,avg1], vm.memory.size[available].

Trigger: Expression attached to Item, returns 0-5 severity levels. Example: last(/host/cpu.load) > 5.

Template: Collection of Items + Triggers + Graphs + LLD, linked to Hosts for batch reuse.

Difference from Prometheus Label Model:

Organization: Zabbix: Host → Item → Trigger hierarchical tree; Prometheus: flat time series, differentiated by labels

Dimensions: Zabbix: multiple Items under a Host, distinguished by key; Prometheus: single series identified by __name__ + labels Slice & Aggregate: Zabbix: weak, cross-host aggregation requires Calculated Items or reports; Prometheus: strong, PromQL aggregates arbitrarily by label

Auto-discovery: Zabbix: LLD discovers multiple instances within a Host; Prometheus: SD discovers multiple Hosts themselves

Configuration: Zabbix: Web UI point-and-click; Prometheus: YAML + Service Discovery

Reuse Unit: Zabbix: Template; Prometheus: Recording Rules + scrape_configs

Fundamental Differences:

Tree vs Flat: Zabbix is "host-centric" tree, cross-host metric aggregation clumsy; Prometheus is "label-dimensional" flat, sum by (job)(rate(...)) aggregates across hosts in one line.

Config as Data vs Config as Code: Zabbix config lives in database, changed via Web UI; Prometheus config is YAML, follows GitOps, enabling version control and audit.

Intuitive vs Flexible: Zabbix tree model intuitive for ops (see all metrics for a machine at a glance) but sacrifices query flexibility; Prometheus sacrifices intuitiveness for PromQL's powerful aggregation.

Reference: Zabbix Full Analysis - IV. Data Model

Q4: Trigger Expression: Disk Free <10% Alert, Require 3-min Average <10%

Expression:

{Linux by Zabbix agent:vfs.fs.size[/,pfree].avg(3m)} < 10

More rigorous with unit constraint (percentage is float):

{host:vfs.fs.size[{#FSNAME},pfree].avg(3m)} < 10

Part-by-part Explanation: {...}: Trigger expression wrapper host: Host name (or host group) vfs.fs.size[/,pfree]: Item key, vfs.fs.size collects filesystem size, parameter / is mount point, pfree means free space percentage .avg(3m): Trigger function, average of values over last 3 minutes (180 seconds) < 10: Threshold, unit is %, so 10 means 10%

Why avg(3m) not last() : last() only checks latest point; disk space transient spikes (temp file write/delete) cause false alerts avg(3m) requires 3-minute average below 10%, filters transient jitter, reduces false positives

Recovery: Trigger auto-recovers — when avg(3m) returns above 10, trigger flips from PROBLEM to OK, fires Recovery operation sending recovery notification.

Advanced — Multi-partition LLD Prototype: Production disks have more than / partition. Use LLD Item Prototype to auto-generate trigger per partition:

{host:vfs.fs.size[{#FSNAME},pfree].avg(3m)} < 10

{#FSNAME}

is LLD macro; discovery rule returns ["{#FSNAME}":"/",{"#FSNAME"":"/data"},...], each entity generates independent trigger.

Reference: Zabbix Full Analysis - IV. Data Model

Q5: LLD vs Prometheus Service Discovery — Same Thing? What Problems Each Solves?

Not the same. They solve different levels of "dynamic" problems.

Discovery Object: Zabbix LLD: multiple monitoring objects within a single Host; Prometheus SD: multiple Hosts themselves

Typical Scenarios: Zabbix LLD: disk partitions, network interfaces, CPU cores, switch ports; Prometheus SD: K8s Pods, cloud ECS, Consul services

Output: Zabbix LLD: multiple Item/Trigger instances; Prometheus SD: scrape target list

Trigger Timing: Zabbix LLD: discovery rule runs on schedule; Prometheus SD: before scrape, during relabel phase

Data Format: Zabbix LLD: JSON array ["{#FSNAME}":"/"]; Prometheus SD: target list + metadata

LLD Solves: A Host contains "multiple homogeneous objects" — one machine has multiple disk partitions ( /, /data, /home), multiple NICs ( eth0, eth1, docker0), multiple CPU cores. Writing an Item per partition manually causes config explosion when partitions change. LLD lets Zabbix auto-discover these objects, using Prototypes to batch-generate Items/Triggers.

Workflow:

1. LLD Rule executes, returns JSON array
   [{"#FSNAME":"/"},{"#FSNAME":"/data"}]
2. For each entity, Item Prototype generates real Item
   vfs.fs.size[/,pfree]
   vfs.fs.size[/data,pfree]
3. Corresponding Trigger Prototype generates real Trigger

Prometheus Service Discovery Solves: Prometheus needs to dynamically know "which targets to scrape /metrics from". In K8s Pods create/destroy constantly, impossible to write static_configs manually. SD lets Prometheus fetch target lists via Kubernetes API, Consul, EC2 API, then relabel to filter and label.

Correspondence:

Zabbix Agent Auto-registration (Agent actively registers to Server) corresponds to Prometheus Service Discovery — both solve "dynamic Host discovery".

Zabbix LLD has no direct counterpart in Prometheus because Prometheus metrics already carry labels (e.g., device=/data), natively supporting multi-partition without "discover then generate Item".

Why This Difference Exists: Zabbix data model is "Host → Item" tree, must have Item before collection; Prometheus is flat label model, application decides which labeled metrics to expose, monitoring side only discovers targets.

Reference: Zabbix Full Analysis - VII. Templates & LLD

Q6: Zabbix Proxy Role? Same Problem as Server HA Cluster?

Not the same problem. They solve different dimensions.

Zabbix Proxy Role — Capacity Scaling: Proxy is a "mini Server" deployed in remote data center, only collects, does not alert:

Cross-data-center partitioned collection: Proxy in East China DC, another in South China, each collects local devices, sends aggregated data back to central Server, saves cross-leased-line bandwidth.

SNMP Partitioning: Distribute 200 network devices across Proxies, avoid single Server polling bottleneck.

Network Isolation: DMZ cannot connect directly to center, Proxy isolates and forwards.

Disconnect Buffer: Proxy buffers data locally when disconnected from Server, replays on reconnect.

Proxy solves "single Server collection capacity bottleneck" — offloads collection load from Server, is horizontal scale-out .

Zabbix Server HA Cluster Role — Disaster Recovery: Since 6.0 LTS Server native HA support:

Two or more Servers share same database

Database maintains ha_node table recording node states

Only one Active node executes collection and alerting at a time, others Standby

Active fails, Standby takes over within seconds

HA solves "Server single point of failure" — Active node crash triggers Standby takeover, is failover disaster recovery , not for scaling. In HA cluster only one node works at a time, total collection capacity does not increase.

Core Differences:

Problem Solved: Proxy: collection capacity bottleneck; HA: Server single point of failure

Goal: Proxy: scale-out horizontal scaling; HA: failover disaster recovery

Database: Proxy: each Proxy local buffer; HA: shared single database

Alert Judgment: Proxy: none (collection only); HA: Active node performs

Simultaneous Work: Proxy: multiple Proxies work concurrently; HA: only one Active works

Production Combination: HA Server (disaster recovery) + multiple Proxies (capacity scaling) + Agent2 (collection) is standard architecture for large-scale Zabbix.

Reference: Zabbix Full Analysis - III. Core Architecture, X. High Availability

Q7: Zabbix vs Prometheus Selection Trade-offs? Common Enterprise Coexistence Patterns?

Selection Trade-offs:

Monitored Objects: Zabbix: physical machines, network devices, storage, IoT; Prometheus: K8s workloads, microservices

Collection Capability: Zabbix: SNMP out-of-the-box, full traditional infrastructure coverage; Prometheus: Micrometer instrumentation + Exporters

Query: Zabbix: SQL-style + trigger expressions; Prometheus: PromQL flexible slice & aggregate

Configuration: Zabbix: Web UI point-and-click, traditional team friendly; Prometheus: YAML + GitOps, DevOps team friendly

Data Model: Zabbix: tree Host-Item-Trigger; Prometheus: flat label dimensions

Long-term Storage: Zabbix: Trends table + TimescaleDB; Prometheus: Thanos / Mimir / VictoriaMetrics

Learning Curve: Zabbix: low (install and use); Prometheus: medium-high (need PromQL + Exporter assembly)

When to Choose Zabbix:

Traditional data center, physical machines, network devices, storage dominant

Heavy SNMP collection for switches/routers/sensors

Team accustomed to Web UI config, unfamiliar with YAML/GitOps

Require official templates out-of-the-box

When to Choose Prometheus:

Cloud-native, K8s workloads dominant

Need flexible multi-dimensional label slicing and PromQL

Strong GitOps / config-as-code requirement

Microservice metric instrumentation (Micrometer)

Common Enterprise Coexistence Pattern:

Physical/Network/IoT --> Zabbix --> \
                                      +--> Unified Alert Platform (DingTalk/Feishu/Webhook)
K8s/Microservices --> Prometheus --> /

Two systems each collect what they excel at:

Zabbix handles traditional infrastructure: Physical OS metrics, switch SNMP, storage arrays, temperature/humidity sensors, legacy Java JMX.

Prometheus handles cloud-native: K8s Pod/Service/Node metrics, Spring Boot Micrometer instrumentation, cAdvisor container metrics.

Alert Convergence: Both systems push alerts via webhook to unified alert distribution layer (custom or open-source AlertRouter/Grafana OnCall), unified noise reduction, inhibition, on-call rotation, escalation.

Why Coexist Not Choose One:

Prometheus poor at network devices ( snmp_exporter requires manual OID mapping, high ops cost)

Zabbix poor at cloud-native (no native K8s SD, Pod up/down tracking clumsy)

Each handles its segment most cost-effectively; forcing unification sacrifices either coverage or flexibility

Reference: Zabbix Full Analysis - XI. Zabbix vs Prometheus Selection

Q8: Production "SNMP Intermittent Timeout Graph Gaps" — Full Troubleshooting and Solution

Symptom: Switch metric curves intermittent, trigger nodata(3m)=1 occasional alerts, curves have gaps.

Troubleshooting Path:

Confirm Timeout Direction:

On Zabbix Server use zabbix_get or snmpwalk to manually fetch device OID, see if timeout reproduces

If manual also times out → network/device side issue; if manual works → Server collection scheduling issue

Check Server Logs: tail -f /var/log/zabbix/zabbix_server.log | grep timeout Watch for SNMPTIMEOUT, retries exceeded, GetNextRequest errors

Check Collection Load:

Check ZBX_STARTSNMPOLLERS configured SNMP Poller count

Check Server internal queues zabbix[queue], zabbix[snmp_busy] for backlog

Estimate NVPS, determine if single Server polling bottleneck

Check Network Layer: ping device IP for latency jitter

Packet capture tcpdump -i any host <device-ip> and udp port 161 to see SNMP request/response round-trip

Check if traversing leased line, whether leased line bandwidth saturated

Check Device Side:

Some switches rate-limit SNMP per source IP

Community string errors or v2c rate-limiting

Device CPU high causing slow SNMP response

Solutions:

Introduce Proxy for Partitioned Collection:

Distribute 200 network devices across multiple Proxies by data center/network segment

Each Proxy handles 1000-2000 NVPS, avoids single Server polling bottleneck

Most thorough solution

Increase SNMP Timeout and Retries:

Server config SNMPTIMEOUT (default 1s) increase to 3-5s SNMPRETRIES (default 3) increase appropriately

But not too large, else single failure drags overall scheduling

Use SNMP v3 Instead of v2c:

Some devices rate-limit v2c, v3 not rate-limited

v3 adds auth+priv, satisfies security compliance

Batch OIDs with Walk:

Single OID fetch round-trip overhead high

Use snmp.walk to fetch entire subtree (e.g., whole ifTable) in one go, reduce round-trips

Zabbix SNMP interface combined with walk for batch collection

Trigger-level Noise Reduction:

Change nodata(3m)=1 to nodata(5m)=1, give SNMP occasional timeout buffer

Or use count(5m,0) > 3 (alert only if >3 losses in 5 minutes), filter transient jitter

Device-side Optimization:

Disable unnecessary SNMP trap alerts on device, reduce device SNMP processing load

Upgrade device firmware, some old firmware have SNMP response bugs

Root Cause Summary: SNMP timeout graph gaps 90% are "single Server polling bottleneck" + "network jitter" combined. Proxy partitioning + v3 + walk batching is the combo; trigger-level noise reduction is just safety net.

Reference: Zabbix Full Analysis - XII. Pitfall Guide

References

Zabbix Full Analysis — companion original article for this interview self-test

Zabbix Official Manual — architecture, collection, triggers, LLD, best practices

Zabbix 6.0 LTS Documentation — Server HA, Agent2, macros, functions

Zabbix Trigger Functions — last / avg / nodata / count

Zabbix LLD Documentation — auto-discovery rules and prototypes

Prometheus Full Analysis — sister article, cloud-native monitoring comparison and selection

Alert System Full Analysis — alert distribution, inhibition, noise reduction methodology