Master Real-Time Hadoop Alerts with Transwarp Manager

Distribution is a key feature of Hadoop; a cluster may consist of hundreds or even thousands of nodes, making performance monitoring a challenging task.

To address this, StarRocks (StarRing) designed and implemented an alert monitoring page in the Transwarp Manager GUI. When any service exceeds a danger threshold, an alert is triggered and displayed on the main alert page for operators.

Three Functional Modules

The alert system is divided into three modules for ease of use: Alert Browsing, Alert Configuration, and Real-time Notification.

Alert Browsing

The browsing interface shows each alert’s trigger time, level, category, status, title, resource, and description. Alerts have two statuses: ACTIVE (unresolved) and CLEARED (resolved). Alerts are classified as automatically cleared (e.g., high CPU load that returns to normal) or requiring manual clearance (e.g., NameNode failover).

Users can filter alerts by category, status (ACTIVE/CLEARED), or time range using the timeline slider.

Alert Configuration

Each service has a dedicated configuration page where users can define alert rules. Configuration items are either non‑numeric (triggered by abnormal behavior) or numeric (threshold‑based). Levels include Warning and Critical . Users can enable or disable alerts, set numeric ranges, and customize thresholds.

Real-time Notification

The system can send alerts via email or invoke custom scripts. Both methods are configurable on the notification page.

Email Notification When an alert fires, the configured recipients receive an email containing details similar to the alert list entry.

Script Invocation After selecting the script trigger option, users place a script (e.g., sendsms.sh ) in /var/lib/transwarp-manager/master/scripts/ . When an alert occurs, the system passes the alert content as the first argument to the script.

Application Examples

NameNode Disk Space Alert

When the NameNode disk usage exceeds 40% of total capacity, a Warning alert is generated; below 15% triggers a Critical alert. The alert page shows the affected node and the specific partition.

Operators should increase partition resources or clean up old files to free space.

Kafka Health Check Alert

The system monitors the percentage of unhealthy Kafka brokers. If more than 5% are unhealthy, a Warning is issued; over 10% triggers a Critical alert.

When such an alert appears, operators should examine the performance metrics of the affected Kafka nodes, check logs, and resolve the underlying issues.

Conclusion

By deploying the TDH alert system, Hadoop users gain real‑time visibility into cluster health, enabling rapid response to potential problems. Mastering the configuration of alert items and setting appropriate thresholds is essential for efficient Hadoop cluster operations.