Build a Scalable Unified Monitoring & Alert Platform with Ganglia & Centreon

Preface

Hello, I am "South Africa Ant" from iLoveLinux. Today I will share how to build a unified operations monitoring platform.

Monitoring is the core of operations; a good platform helps detect problems early and notify responsible personnel.

Effective operations rely on precise, comprehensive monitoring of business systems to discover issues promptly.

Problems are not scary; the real issue is not detecting them until customers notice failures, which a monitoring platform can prevent.

Design Concept of a Unified Monitoring & Alert Platform

The platform focuses on runtime monitoring and fault alerting, integrating network, hardware, software, and database resources from all business systems.

It achieves unified management, standardization, processing, presentation, single sign‑on, and unified permission control, enabling standardized, automated, and intelligent operations.

Six Layers of the Intelligent Operations Monitoring Platform

The architecture consists of six layers:

Data Collection Layer: Collects network, business, database, and OS data, normalizes it, and stores it.

Data Presentation Layer: Web interface that visualizes collected data as charts to help operators understand system status and trends.

Data Extraction Layer: Filters and extracts needed data for the monitoring & alert module.

Alert Rule Configuration Layer: Sets alert thresholds, contacts, and methods based on extracted data.

Alert Event Generation Layer: Records alert events, stores results in a database, and generates analysis reports.

User Presentation Management Layer: Top‑level web UI that displays monitoring and alert results with multi‑user, multi‑permission management.

Three Main Modules

The six layers are grouped into three functional modules:

Data Collection Module: Gathers basic metrics via SNMP, agents, or custom scripts; this article uses Ganglia.

Data Extraction Module: Filters and forwards required data to the monitoring & alert module.

Monitoring & Alert Module: Configures monitoring scripts, alert rules, thresholds, contacts, and visualizes alert results; tools include Nagios and Centreon.

Ganglia as the Data Collection Module

Ganglia is a scalable distributed monitoring system for HPC clusters. Each node runs a gmond daemon that collects CPU, memory, disk, I/O, and network metrics, sending them to a gmetad daemon which stores data with RRDTool and presents it via a PHP web page.

Key Features of Ganglia

Flexible distributed, hierarchical architecture supporting thousands of nodes and easy regional deployment.

Accurate real‑time and historical data, enabling performance tuning, upgrades, and capacity planning.

Supports both multicast and unicast data collection, reducing load in large deployments.

Collects CPU, memory, disk, I/O, process, and network metrics; extensible via C or Python plugins for custom data.

Ganglia provides many ready‑made Python modules: https://github.com/ganglia/gmond_python_modules

Compared with Cacti/Zabbix, Ganglia offers better scalability and real‑time accuracy for large‑scale environments.

Centreon as the Monitoring & Alert Module

While Ganglia gathers data, operators need automated alerting. Centreon provides a powerful distributed IT monitoring system built on top of Nagios, offering:

Open‑source availability.

Deep integration with Nagios (or its own core) for data collection.

Web‑based configuration and management of Nagios.

Support for plugins such as NRPE, SNMP, and NSClient.

Centreon stores monitoring data in a database, reads it in real time, and presents it via a web UI with multiple notification channels (SMS, email, custom scripts).

Seamless Integration of Ganglia and Centreon

Ganglia excels at data collection; Nagios excels at alerting. Combining them leverages each tool’s strengths:

Ganglia lacks built‑in alerting; Nagios provides it.

Nagios lacks distributed data collection; Ganglia provides it.

Ganglia offers strong reporting; Nagios can use plugins for visualization.

Ganglia’s APIs allow feeding its metrics into Nagios.

The integration requires a data extraction module to pull metrics from Ganglia and feed them to Centreon/Nagios. This can be done with custom scripts (PHP or Python) or existing scripts such as check_ganglia_metric.php and check_ganglia_metric.py.

Overall System Architecture

The platform consists of:

Distributed Ganglia nodes (gmond) in each data center, reporting to a Ganglia proxy (gmetad).

Manager server that runs the data extraction module, consolidates data, and hosts Centreon.

High‑availability setup with a backup manager server.

Web UI that unifies Ganglia and Centreon dashboards.

Data flow: gmond → multicast/unicast → gmetad (XML over TCP) → RRD storage → Centreon extracts metrics → alert generation.

Ganglia Data Flow Diagram

gmond collects local metrics, exchanges data with peers via UDP (XDL format), gmetad pulls XML data from gmond/gmetad, updates RRD databases, and Centreon reads the data for alerting and visualization.

Q&A

Is gmetad a single node? What configuration is needed for 10,000 hosts?

gmetad should be deployed in at least a primary‑secondary pair; a distributed structure across three data centers is recommended.

What alert strategies are supported?

Centreon supports flexible rules: greater than, less than, equal, retry counts, intervals, etc.

What monitoring items are needed for middleware and databases?

Customizable based on business needs; typically system metrics, business‑level metrics without logical relationships, and metrics with logical relationships.

How does gmond transmit data?

gmond uses UDP for both unicast (upstream reporting) and multicast (peer collection); data is formatted in XDL.

Does the platform support Windows and Linux?

Yes, it supports full‑platform monitoring for various OS, network devices, and switches.

Is there a dedicated team maintaining the platform?

Yes, a dedicated operations team handles maintenance and further development.

Can the platform monitor fluctuations?

Ganglia’s UDP‑based, high‑frequency data collection makes it suitable for monitoring rapid fluctuations across distributed sites.

How many monitoring items and alerts are generated for 10,000 servers?

Over 10,000 monitoring items; each data pull generates 20‑30 MB of XDL data, focusing on Hadoop metrics.

How to handle long‑running monitoring checks and resource constraints?

Alert latency is kept around 10 seconds; gmond’s low overhead minimizes impact. High‑performance CPUs and disks are needed at aggregation points, and timeout mechanisms can be set for slow checks.

Team size?

Six people: 2 developers/ops, 3 system (business) ops, 1 network ops.