How to Implement Self‑Monitoring for Your Monitoring System with Prometheus and Catpaw

Problem

Monitoring platforms are critical services (P0) and must also be monitored. Direct self‑monitoring can create circular dependencies, while deploying a completely separate monitoring stack adds unnecessary complexity.

Solution 1 – Use the system’s own metrics

Most monitoring systems (e.g., Prometheus, VictoriaMetrics, Nightingale) expose internal metrics at a /metrics HTTP endpoint. By configuring the same monitoring platform to scrape this endpoint, you can store historical trends and define alerts. As long as at least one component of the monitoring stack remains operational, the metrics stay available.

Example – Nightingale with the categraf input.prometheus plugin :

[[instances]]
urls = [
    "http://localhost:17000/metrics"
]

Replace localhost:17000 with the actual Nightingale address. After adding the configuration, import the built‑in dashboard from the repository:

https://github.com/ccfos/nightingale/tree/main/integrations/n9e/dashboards

Solution 2 – Alive monitoring with an external lightweight monitor

If several modules of the monitoring system fail simultaneously, internal metrics may become unavailable and the alert engine itself could be down. In such scenarios a separate lightweight monitor should verify the liveness of critical components and use an independent notification channel.

One practical combination is catpaw (v0.7.0) together with an external SaaS alerting service (e.g., FlashDuty). catpaw provides a variety of plugins; the most relevant for self‑monitoring are: net – probes TCP/UDP ports. procnum – checks the number of running processes.

Port‑level check (net plugin)

[[instances]]
targets = [
    # "127.0.0.1:22",
    # "localhost:6379",
    # ":9090"
]

# Timeout for the connection (default 5s)
timeout = "5s"
# Read timeout for responses (if applicable)
read_timeout = "5s"
# Number of concurrent probes per instance
concurrency = 10
# Collection interval
interval = "30s"
# Optional static labels
labels = { env="production", team="devops" }

# Protocol must be "tcp" or "udp"
protocol = "tcp"
# For TCP checks you can optionally send/expect strings
send = "ssh"
expect = "ssh"

[instances.alerting]
enabled = true
for_duration = 0               # equivalent to Prometheus "for"
repeat_interval = "5m"
repeat_number = 3
recovery_notification = true
default_severity = "Warning"

If a target IP:Port is unreachable, an alert is generated according to the [instances.alerting] section.

Process‑level check (procnum plugin)

[[instances]]
# search_cmdline_substring = ""   # optional pattern for pgrep -f
alert_if_num_lt = 1               # trigger when process count < 1
check = "进程存活检测(进程数量检测)"
interval = "30s"

[instances.alerting]
enabled = true
for_duration = 0
repeat_interval = "5m"
repeat_number = 3
recovery_notification = true
default_severity = "Warning"

Combining the net and procnum plugins allows the external monitor to detect severe failures (process crash or port outage), while routine issues remain visible through the monitoring system’s own metrics.