Mastering Alert Management with Nightingale: Rules, Silencing, Escalation, and Self‑Healing

Monitoring is a method, alerting is a means, solving is the goal.

Many users collect numerous metrics but struggle to decide which should trigger alerts, how to route those alerts to the right teams, and how to handle escalation.

With Prometheus+Alertmanager, I used DingTalk groups and tags to route alerts, but time‑based escalation was cumbersome.

Nightingale simplifies alert rule management and does it elegantly.

Alert Rules

Before creating alerts, identify the metrics that need monitoring at system, application, and business levels (CPU, memory, disk, IO, saturation, failure rate, latency, transaction failures, etc.).

Nightingale offers built‑in rules and custom rules.

Built‑in rules provide generic templates; they become active only after being cloned into your rule set. For example, I cloned the Linux

TIME_WAIT

rule into the default business group.

After cloning, the rule appears in the rule overview.

You can create multiple business groups to separate alerts for front‑end and back‑end teams.

Imported rules are inactive by default and require configuration.

The data source is

local_prometheus

, indicating the originating cluster.

The condition triggers when the total

TIME_WAIT

exceeds 20000.

The alert level is set to level 2 (moderate importance).

The evaluation interval is every 15 seconds; if the condition persists for 60 seconds, the alert fires.

Additional settings include effective time windows, business group applicability, and notification channels.

Enable recovery notifications to inform owners when an alert clears.

Specify the receiving business group.

Set a watch‑time after recovery before sending a notification, reducing noise from flapping alerts.

Configure repeat notifications for unresolved alerts (note: this does not handle escalation).

Silencing Alerts

Non‑critical alerts can be silenced, for example during a deployment window, by creating a silencing rule for a specific business group.

Active alerts can also be silenced with a single click, and historical alerts can be muted similarly.

Alert Escalation

If an alert remains unaddressed for a certain period, you can upgrade its severity and route it to a higher‑level group.

In Nightingale, escalation is configured in the subscription rules, e.g., upgrading a

notice

alert to level 1 after one hour and sending it to a senior group.

Self‑Healing

Repetitive operational tasks can be automated with self‑healing scripts to improve efficiency and reduce human error.

Nightingale’s self‑healing uses the

ibex

template. The

ibex-server

must be deployed separately, while the

ibex-agent

is integrated into Categraf.

Deploy ibex‑server

Download the binary from the GitHub releases page. The package contains the executable, a tarball, and an

etc

directory.

<code># ll
total 21536
drwxr-xr-x 3 root root 4096 Apr 19 10:44 etc
-rwxr-xr-x 1 root root 16105472 Nov 15 2021 ibex
-rw------- 1 root root 5931963 Jun 3 2022 ibex-1.0.0.tar.gz
drwxr-xr-x 2 root root 4096 Nov 15 2021 sql
</code>

Import the database schema:

<code>mysql -uroot -p &lt; sql/ibex.sql</code>

Edit

/etc/server.conf

to set database connection details, then start the server:

<code>nohup ./ibex server &gt; server.log &amp;</code>

Configure the Client

In the system settings → notification settings → alert self‑healing module, set the server address.

Test Self‑Healing

Add a script in the self‑healing → script section, save, and create a task. You can execute it immediately.

During testing I encountered several issues: missing pre‑conditions for ibex‑server and ibex‑agent, lack of detailed failure logs, and an unintuitive placement of the self‑healing entry in N9e V6.

Ultimately the front‑end timed out and the back‑end produced no logs.

Conclusion

Nightingale provides comprehensive alert rule management, channel distribution, suppression, and escalation, and FlashDuty can integrate alerts from multiple clusters, which is sufficient for most enterprises.

However, I was unable to successfully test the self‑healing feature, likely due to environment-specific factors such as Helm deployment of N9e versus binary deployment of ibex‑server.