From Firefighting to Fire Prevention: Production-Grade Database Monitoring with Prometheus & Grafana

Scope

Applies primarily to MySQL 8.0/5.7 with Redis; suitable for Kubernetes, VM, and bare-metal hybrid deployments. Component versions, image tags, and metric names must match verified versions; avoid using latest in production.

Abstract

True reliable database monitoring is not merely having a Grafana dashboard, but a continuously closed-loop engineering system spanning risk signal collection, semantic computation, tiered alerting, event correlation, remediation runbooks, and retrospective improvement. It must answer five questions: Is the database available? Is performance degrading? Is data correct and fresh? How long until capacity exhaustion? When an alert fires, who responds, within what time, and via what steps?

The article presents a progressively adoptable solution: Prometheus handles collection, rule evaluation, and short-term hot storage; Alertmanager manages alert routing and noise reduction; Grafana provides visualization and diagnostic entry points; larger scales extend via remote write and long-term storage. The focus is not on piling metrics but on establishing operable SLI/SLO and failure loops.

1. Starting from a Replication Lag Incident: Why Dashboards Can't Save You

A transaction database once suffered an incident: during peak traffic, an unindexed UPDATE ran long on a replica, causing replication SQL thread backlog. Writes to primary remained normal; monitoring showed no QPS or CPU anomalies. Only when users read stale order status from the replica did the team discover replication lag had jumped from sub-second to tens of seconds.

Such incidents typically expose four design gaps:

Collecting only resource metrics, not user impact. CPU and connection counts are symptoms; reading stale data or order failures are service risks.

Setting only thresholds, not detecting failure modes. Monitoring only Seconds_Behind_Master misses replication thread stops, metric collection failures, and upstream primary unavailability.

Having only notifications, no remediation context. On-call receives "lag 47 seconds" but must still scramble for instance details, topology, change records, and diagnostic commands.

No feedback loop. Every incident relies on manual firefighting; thresholds, runbooks, capacity models, and release gates are never fed back.

The goal of a production-grade system is to turn "discovering at 3 AM" into "risk identified, aggregated, routed, and handled by a defined process the moment it appears."

2. Define Success First: SLI, SLO, and Alert Boundaries

2.1 Database Is More Than "Online"

Model signals in four categories, each mapping to distinct business impact and remediation paths:

Availability: Can necessary access be established and completed? Typical SLI: mysql_up, probe success rate. Example SLO: Core database monthly availability ≥ 99.95%.

Performance: Are requests fast enough? Typical SLI: P95/P99 query latency, lock wait, connection pool wait. Example SLO: Core read path P99 < 100 ms.

Correctness & Freshness: Will we read stale, wrong, or lose data? Typical SLI: Replication thread status, replication lag, backup recoverability. Example SLO: Read replica freshness < 10 s.

Capacity & Resilience: Is exhaustion imminent? Can we recover from failure? Typical SLI: Disk prediction, connection headroom, backup age, recovery drill results. Example SLO: Disk not exhausted within 14 days; backup RPO < 24 h.

Internal database metrics cannot fully replace end-to-end SLIs. For example, mysql_up=1 only proves exporter connectivity, not that application accounts, network paths, DNS, connection pools, or critical SQL are functional. Critical paths should also expose connection pool wait, SQL latency, and key transaction success rates from the application side, with synthetic probes where needed.

2.2 Alerting Is Not a Synonym for All Anomalies

Alerts should have three attributes: require human intervention, handling within an acceptable window adds value, and carry context needed for remediation . Otherwise they belong on dashboards as trends, capacity tickets, or log events—not as 3 AM pagers.

A practical tiering model:

P1 / Critical: User impact or imminent impact; no redundancy. Notification: immediate on-call page. Target: 5 min acknowledge, 30 min mitigation.

P2 / Warning: Clear risk but remediation window exists. Notification: aggregated on-call notification or ticket. Target: handle during work hours or continuous follow-up.

Info: Capacity, config drift, low-priority trends. Notification: dashboard / weekly report. Target: planned governance.

Do not set "high QPS" directly as critical. High QPS may be normal promotion traffic; it only represents risk when combined with error rate, latency, connection saturation, or baseline deviation.

3. Principles: Why Prometheus Can Monitor and Why It Slows

3.1 Pull Model and Time Series

Prometheus periodically scrapes exporter /metrics endpoints per configuration. A time series is uniquely identified by metric name plus full label set:

mysql_global_status_threads_connected{cluster="order",instance="mysql-01:9104",role="primary"}

Labels like job, instance, cluster, env are stable and bounded and suitable as dimensions. High-cardinality values like order IDs, SQL text, user IDs, table names, full error messages must not enter labels; they belong in logs, slow-query platforms, or tracing systems.

Prometheus's local TSDB writes new samples to a Head block, then compacts into immutable blocks; indexing supports label filtering. It excels at "recent aggregation and alerting" but is not an arbitrary high-cardinality analytical database. Active series growth increases memory, WAL, compaction, and query overhead; slow queries often stem from overly wide time ranges or large label match sets.

3.2 Four Common Misconceptions

up differs from mysql_up . up indicates Prometheus can scrape the exporter; mysql_up indicates exporter can connect to the database. Both must be watched.

"Lag = 0" does not equal healthy replication. When threads stop, the lag field may be null; must check IO/SQL (or receiver/applier) thread states and replication errors.

Averages mask spikes. For latency, prefer P95/P99 or histogram quantiles; averages only assist judgment.

Scraping faster is not better. 15–30 seconds suits database base metrics; shorter intervals amplify collection cost and remote write pressure, and cannot replace application-side high-precision tracing.

4. Target Architecture: Decoupling Collection, Computation, Notification, Query

Architecture layers (left to right): MySQL/Redis → Exporter/Custom Collector → Node Exporter → Prometheus A / Prometheus B → Alertmanager Cluster → remote_write → Thanos/VictoriaMetrics/Mimir → Grafana → Phone/IM/Ticket → Runbook, Change & Topology Links.

4.1 Layer Responsibilities

Collection Layer: Convert database state to metrics. Production points: least privilege, TLS, pinned versions, scrape timeouts, target health checks.

Prometheus Layer: Scrape, short-term storage, rule evaluation. Production points: dual replicas, reasonable retention, rule grouping, resource isolation.

Alerting Layer: Deduplication, grouping, inhibition, routing. Production points: HA, on-call routing, maintenance silences, runbook URLs.

Long-term Storage/Query Layer: Long retention, cross-cluster query, downsampling. Production points: remote write backpressure observability, tenant/retention policies, cost governance.

Presentation & Process Layer: Dashboards, diagnostics, retrospectives. Production points: dashboards as entry points; changes, topology, runbooks clickable.

4.2 HA Is Not "Deploy Two"

Two Prometheus replicas scraping the same target tolerate single-instance failure but produce duplicate samples. Alertmanager cluster deduplicates; query layer needs replica-deduplication capability (e.g., long-term storage) or Grafana must explicitly query a single replica. Replicas should span availability zones with independent failure domains, persistent disks, and resource quotas.

Dual scraping does not solve exporter, database, network path, or same-datacenter simultaneous failures; those require database HA, cross-AZ deployment, and application degradation strategies.

5. Metric Design: From "Collectable" to "Decidable"

5.1 MySQL Minimum Viable Metric Set

Organized by failure mode:

Instance unreachable: up, mysql_up. Diagnostics: TCP, DNS, app probe, recent releases.

Connection exhaustion: Threads_connected / max_connections, Threads_running, app pool wait. Diagnostics: idle connections, connection creation rate, anomalous clients.

Query slowdown: App SQL P95/P99, slow query rate. Diagnostics: QPS, temp tables, full scans, execution plans.

Lock contention: Row lock waits, long transactions, processlist. Diagnostics: deadlock increments, blocking chains, transaction age.

InnoDB pressure: Buffer pool hit ratio, dirty pages, log waits, history list length. Diagnostics: disk IOPS/latency, checkpoint pressure.

Replication anomaly: Thread states, lag, error codes, GTID diff. Diagnostics: replica read-only, relay log, long transactions.

Capacity risk: Data disk usage, predicted exhaustion time, binlog growth. Diagnostics: table growth, backup availability, scaling window.

Recoverability: Last successful backup time, checksum, drill results. Diagnostics: RPO/RTO, backup chain integrity.

Metric names vary with exporter version, MySQL version, and enabled collectors. Before deployment, hit the exporter's /metrics and write rules against actual metric names; never copy rules assuming they work.

5.2 Label Specification and Cardinality Budget

Recommended global external labels: env, region, cluster, service, role. instance is usually auto-attached by Prometheus. Label values should come from controlled enums; do not add Pod UIDs, container IDs, dynamic shard IDs, SQL fingerprint full text, etc., to common metrics.

Treat "active series count, ingestion rate, rule evaluation duration, remote write queue length" as monitoring platform SLIs. Any new exporter or collector must pass a pre-launch review stating: expected new series per instance, max label cardinality, scrape duration, and failure degradation behavior.

6. Secure Deployment: Accounts, Secrets, and Exporter Deployment

6.1 Least-Privilege Account (MySQL Example)

Permissions below suit common status and replication collection; exact privileges must be validated against exporter docs, enabled collectors, and MySQL version. Passwords delivered via secret management; no real passwords in examples.

CREATE USER 'mysqld_exporter'@'10.%' IDENTIFIED BY 'REPLACE_WITH_SECRET';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysqld_exporter'@'10.%';
-- MySQL 8.0 for replication_* performance_schema views, grant as needed:
GRANT SELECT ON performance_schema.* TO 'mysqld_exporter'@'10.%';

Restrict source CIDR, prefer TLS with certificate verification. Never grant SUPER, ALL PRIVILEGES, reuse business accounts, or store plaintext credentials in Compose, Helm values, Git repos, or Grafana annotations.

6.2 VM/Bare Metal: Standalone Exporter

Run exporter as systemd service or controlled container, listening only on address open to Prometheus CIDR; database accessed via loopback or private network. Standalone exporter decouples from database lifecycle, easing upgrades, rate limiting, and auditing. Sidecar suits database-as-Pod scenarios where shared Pod network namespace brings clear convenience; it is not the default for all stateful services. config.my-cnf (permissions 0600) example:

[client]
user=mysqld_exporter
password=REPLACE_WITH_SECRET
host=127.0.0.1
port=3306

Startup args example (verify names per image version):

mysqld_exporter 
  --config.my-cnf=/etc/mysqld_exporter/config.my-cnf 
  --web.listen-address=0.0.0.0:9104 
  --collect.global_status 
  --collect.global_variables 
  --collect.info_schema.innodb_metrics

Do not blindly enable all collectors. performance_schema, table-level collectors, or custom queries may add SQL overhead and high cardinality; measure single-scrape duration and new series first, then enable incrementally.

6.3 Kubernetes: Declarative Collection with Secret Mounting

With Prometheus Operator, use ServiceMonitor to describe targets; credentials via Secret mount or controlled injection. Below shows target discovery only; TLS, NetworkPolicy, ServiceAccount, and Secret fields must be completed per cluster policy:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mysql-exporter
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: mysqld-exporter
  endpoints:
  - port: metrics
    interval: 15s
    scrapeTimeout: 10s
    path: /metrics

scrapeTimeout

must be less than interval. If scrapes frequently timeout, first diagnose exporter SQL, database load, or network issues—not simply raise timeout.

7. Prometheus: Maintainable Collection, Recording, and Alerting Rules

7.1 Collection Config

Non-Kubernetes example. external_labels used for cross-cluster remote write, alert attribution, and query deduplication; relabel_configs standardize controlled labels. Actual addresses should come from service discovery, not large static lists.

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    env: production
    region: cn-east-1
    prometheus_replica: prom-a

rule_files:
  - /etc/prometheus/rules/mysql-recording.yml
  - /etc/prometheus/rules/mysql-alerts.yml

scrape_configs:
  - job_name: mysql
    file_sd_configs:
      - files: [/etc/prometheus/targets/mysql/*.json]
        refresh_interval: 1m
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

file_sd

files should be generated by CMDB, deployment system, or controlled controller—not manually maintained long-term. Service registration events must be traceable: who added instance, which cluster, primary/replica role, when decommissioned.

7.2 Recording Rules: Move High-Frequency Computation Forward

Recording rules reduce dashboard and alert query cost and unify calculation semantics. Rules below intentionally keep few labels to avoid passing meaningless dimensions to aggregated metrics; verify metric names and units before release.

groups:
- name: mysql-recording
  interval: 30s
  rules:
  - record: mysql:connections:utilization_ratio
    expr: |
      mysql_global_status_threads_connected
      / on (instance) mysql_global_variables_max_connections
  - record: mysql:queries:rate5m
    expr: rate(mysql_global_status_queries[5m])
  - record: mysql:slow_queries:rate5m
    expr: rate(mysql_global_status_slow_queries[5m])

For business "dynamic baselines", do not directly compare current value to 7-day average: periodic workloads cause false positives. At minimum bucket by weekday/hour, or use mature anomaly detection; run baseline alerts as warning first, validate recall and noise before promoting.

7.3 Production Alerting Rules: Coverage, Duration, Context

Rules use generic PromQL. Replication metric exact names differ across MySQL 8 and exporter collectors, so abstract recording metrics mysql:replication_healthy and mysql:replication_lag_seconds are used as examples: first normalize actual source/replica metrics to these in an adaptation layer, then alerts and dashboards stay stable across versions.

groups:
- name: mysql-alerts
  rules:
  - alert: MySQLExporterUnreachable
    expr: up{job="mysql"} == 0
    for: 2m
    labels:
      severity: warning
      service: database
    annotations:
      summary: "MySQL exporter unreachable: {{ $labels.instance }}"
      description: "Prometheus unable to scrape exporter for 2 minutes; differentiate exporter, network, and database faults."
      runbook_url: "https://runbooks.example.com/mysql/exporter-unreachable"

  - alert: MySQLUnavailable
    expr: mysql_up == 0
    for: 1m
    labels:
      severity: critical
      service: database
    annotations:
      summary: "MySQL unavailable: {{ $labels.instance }}"
      description: "Exporter connection failed for 1 minute; check database process, network, auth, and host resources."
      runbook_url: "https://runbooks.example.com/mysql/unavailable"

  - alert: MySQLConnectionSaturation
    expr: mysql:connections:utilization_ratio > 0.85
    for: 5m
    labels:
      severity: warning
      service: database
    annotations:
      summary: "MySQL connection utilization over 85%: {{ $labels.instance }}"
      description: "Current ratio={{ $value | humanizePercentage }}. Check app pool leaks, slow SQL, anomalous clients."
      runbook_url: "https://runbooks.example.com/mysql/connection-saturation"

  - alert: MySQLReplicationUnhealthy
    expr: mysql:replication_healthy == 0
    for: 1m
    labels:
      severity: critical
      service: database
    annotations:
      summary: "MySQL replication thread abnormal: {{ $labels.instance }}"
      description: "Replication health check failed; do not rely solely on lag field."
      runbook_url: "https://runbooks.example.com/mysql/replication"

  - alert: MySQLReplicationLagHigh
    expr: mysql:replication_lag_seconds > 30
    for: 5m
    labels:
      severity: warning
      service: database
    annotations:
      summary: "MySQL replication lag persistently high: {{ $labels.instance }}"
      description: "Lag={{ $value }}s for 5 minutes. Check large transactions, lock waits, disk, network."
      runbook_url: "https://runbooks.example.com/mysql/replication-lag"

  - alert: MySQLDiskWillFillSoon
    expr: |
      predict_linear(node_filesystem_avail_bytes{mountpoint="/var/lib/mysql"}[6h], 14 * 24 * 3600) < 0
    for: 1h
    labels:
      severity: warning
      service: database
    annotations:
      summary: "MySQL data disk predicted to fill within 14 days: {{ $labels.instance }}"
      runbook_url: "https://runbooks.example.com/mysql/disk-capacity"

Rule validation is a release step, not optional:

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/mysql-alerts.yml

In staging, replay historical data or run short-term shadow to verify: rules load, actual metrics exist, labels join correctly, expected failures trigger, normal peaks don't false alert.

8. Alertmanager: Turning Alerts into Actionable Events

8.1 Routing, Grouping, Inhibition

When a database becomes unreachable, exporter scrape failure, mysql_up=0, replication break, and app error rate spike often fire together. On-call needs one "database unavailable" event, not five notifications.

route:
  receiver: default-notify
  group_by: [alertname, cluster, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
  - matchers:
    - severity="critical"
    receiver: oncall-im
    repeat_interval: 30m
  - matchers:
    - severity="warning"
    receiver: database-team

inhibit_rules:
  - source_matchers:
    - alertname="MySQLUnavailable"
    target_matchers:
    - service="database"
    - severity="warning"
    equal: [cluster, instance]

Inhibition rules must be validated via failure drills; avoid accidental matches from alerts missing instance or cluster labels. Maintenance windows use silences with ticket number, owner, and expiry; never permanently disable rules for "quiet".

8.2 What a Qualified Notification Must Contain

Notification template at minimum: alert name & severity, environment/cluster/instance/role, start time & duration, current value & threshold, Grafana deep link, runbook link, CMDB/topology link, associated change ID. Links generated from controlled labels; avoid interpolating unvalidated labels into URLs.

9. Grafana: From Display Tool to Diagnostic Workbench

Build in three layers—"Overview → Instance → Specialty"—not everything on one page:

Database Service Overview: Instance availability, error budget, replication health, capacity prediction, current alerts; for on-call to quickly assess blast radius.

Instance Diagnostic Page: QPS, connections, threads, latency, locks, buffer pool, disk, network, recent changes; for bottleneck localization.

Specialty Pages: Replication, slow SQL, lock waits, backup/restore, capacity; for domain teams to deep-dive.

Every panel needs units, reasonable min step, clear threshold colors, and current value—not just lines. Critical panels must provide variables: env, region, cluster, instance, role, defaulting to aggregated view with drill-down. Dashboard JSON, datasources, and alert templates in Git, deployed via provisioning or IaC; no untraceable manual edits.

Recommend data links on panels: from instance to CMDB, from replication lag to runbook, from slow query spike to log/tracing platform. Thus alert page becomes diagnostic entry, not dead end.

10. High Concurrency & Scalability: Capacity Model and Evolution Path

10.1 Estimate First, Don't Just Add Machines

Approximate active series count:

instance_count × series_per_instance × label_combination_multiplier

Approximate ingestion rate: active_series / scrape_interval Not a precise capacity formula, but sufficient to spot two risks: a collector creating series per table/user/SQL fingerprint; or scrape interval unjustifiably reduced from 30s to 5s. Pre-launch load test and observe Prometheus self-metrics: head series, TSDB WAL, rule eval duration, query latency, remote write pending/failed samples.

10.2 Phased Architecture

Start: Few instances, short retention → Single Prometheus + local disk. Governance: rule quality, backup, basic alerts.

HA: Monitoring SPOF not allowed → Two Prometheus + Alertmanager cluster. Governance: replica deduplication, cross-AZ, failure drills.

Long Retention: Need monthly/quarterly trends → remote_write + long-term storage. Governance: retention policies, write backpressure, cost.

Multi-cluster: Multi-region/team → Sharded collection + global query layer. Governance: tenant isolation, global labels, permissions.

Thanos, VictoriaMetrics, Mimir all serve long-term storage/unified query; choose based on existing object storage, ops capability, query isolation, multi-tenancy, cost—not just single-node write benchmarks. Remote write isn't free: network outage or slow receiver causes local WAL/queue buildup; must monitor and set local disk headroom and degradation strategy.

10.3 Remote Write Example and Backpressure Governance

remote_write:
  - url: https://metrics-write.example.com/api/v1/write
    # Use controlled credentials or mTLS; no plaintext tokens in config
    queue_config:
      capacity: 20000
      max_samples_per_send: 2000
      min_shards: 1
      max_shards: 32

Don't blindly copy large max_shards or capacity. These need tuning against write rate, network bandwidth, receiver throughput, and Prometheus memory pressure tests; on sustained backpressure, prioritize investigating receiver, network, and cardinality growth before scaling or adjusting queues.

11. Real-World Scenario: Replication Lag from Alert to Mitigation

11.1 Event Flow

App → On-call ← Alertmanager ← Prometheus ← Exporter ← Replica
Replication state / lag / error → /metrics → rule sustained 5 min → MySQLReplicationLagHigh
Grouped by cluster, routed to on-call
Check threads, errors, large transactions, IO/disk
If needed, route strong-consistency reads back to primary or degrade
Remove blockage, restore replication, verify catch-up

11.2 Runbook: Replication Lag High

0–5 minutes: Confirm Impact & Scope

Confirm alerted instance role, cluster, duration, lag trend, and whether MySQLUnavailable also firing.

Check replication thread states and recent errors (MySQL 8: SHOW REPLICA STATUS\G; 5.7: SHOW SLAVE STATUS\G). Verify IO and SQL/Applier threads, not just lag field.

In Grafana, correlate primary write rate, replica disk latency, network, long transactions, lock waits, recent releases/DDL.

5–30 minutes: Mitigation & Localization

If stale reads impact core flows: per predefined plan, route strong-consistency reads to primary, use consistency routing, or temporarily disable problematic read path; must assess primary capacity.

If SQL/Applier blocked by large transaction, DDL, or lock: identify transaction ownership and business impact, then DB owner executes controlled kill, rate limit, or change rollback. Never KILL without understanding transaction semantics.

If IO thread or network abnormal: rule out primary unavailability, credential changes, TLS/network policy, binlog purge; after fix, confirm replication reconnects and lag monotonically decreases.

Recovery Judgment & Retrospective

Close event only when replication threads healthy, lag continuously stable within business threshold, read paths restored, and app errors/latency back to baseline. Retrospective must capture: triggering SQL/change, detection timeliness, alert noise, mitigation duration, automation opportunities, and needed release checks or index governance.

12. Common Pitfalls and Correct Practices

Anti-pattern: Exporter uses root or business account. Why dangerous: Excessive privileges, audit difficulty. Fix: Least privilege, restrict source, Secret/mTLS.

Anti-pattern: Passwords in Compose/Git. Why dangerous: Leakage hard to trace/rotate. Fix: Secret management, file mount, rotation process.

Anti-pattern: Only monitor replication lag. Why dangerous: Thread stop may show null or misleading. Fix: Joint judgment of thread state, error, lag, app freshness.

Anti-pattern: Per-table/per-SQL labels. Why dangerous: Cardinality explosion, TSDB OOM. Fix: Metrics keep controlled dimensions; details go to logs/tracing.

Anti-pattern: Alert at 50% connection usage. Why dangerous: Peak noise drowns real faults. Fix: Ratio + duration + pool wait/business latency.

Anti-pattern: Enable all exporter collectors. Why dangerous: Collection SQL may hurt production. Fix: Enable on demand, load-test scrape duration and series count.

Anti-pattern: Two Prometheus = HA done. Why dangerous: Still possible dual-write, dual-alert, same failure domain. Fix: Alertmanager HA, deduplication, cross-AZ, drills.

Anti-pattern: Alerts without runbooks. Why dangerous: Notification only transfers anxiety. Fix: Every P1/P2 has explicit acknowledge, diagnose, mitigate, recovery conditions.

13. Release, Verification, and Continuous Operations Process

13.1 Safe Rule Release Pipeline

Requirement: Failure mode/SLI → Design: Metrics, labels, thresholds, runbook → Pre-release: Collection & rule validation → Shadow: Display only, no notify → Canary notify: Limited routing → Full release → Drill, retrospective, threshold calibration

Pre-launch checklist:

Metrics exist on target exporter, units and labels verified. promtool config and rule validation pass; rule eval duration below evaluation interval.

New series count, scrape duration, DB collection SQL overhead within budget.

Alerts include owner, severity, env, instance, Grafana and runbook links.

Alerts validated in test failure, recovery, and maintenance silence scenarios.

Dashboards, rules, routing, contacts managed by Git/IaC, rollbackable.

13.2 Operational Cadence

Daily: Check unresolved P1/P2 alerts, collection failures, remote write backpressure.

Weekly: Audit alert volume, acknowledge rate, false positive rate, MTTA, MTTR, noisiest rules.

Monthly: Review capacity predictions, backup success rate and restore drills; clean expired silences, targets, dashboards.

Quarterly: Drill database unavailability, replication break, remote storage outage, Alertmanager failure; verify RPO/RTO.

Institutionalize "decommission or refactor ineffective alerts": alert quality is not one-off config but a continuously maintained asset like application code.

14. Maturity Model and 30-Day Landing Path

L0 – Reactive Firefighting: SSH debug after user complaints. Next: Establish four basic signals—instance availability, connections, replication, disk.

L1 – Has Dashboards: Charts but no on-call loop. Next: Configure tiered notifications and minimal runbooks.

L2 – Alertable: Many alerts, slow triage. Next: Inhibition, correlation, duration, alert quality metrics.

L3 – Predictable: Capacity and trends visible. Next: Exhaustion prediction, baseline analysis, long-term storage, cost governance.

L4 – Resilient Operations: Drills, automation, retrospective-driven improvement. Next: Automated diagnosis/degradation, codify experience into release gates.

Week 1 – Foundation. Map database topology and business tiers, establish exporter least privilege and target discovery, ingest up, mysql_up, connections, replication, disk, backup signals.

Week 2 – Close the Loop. Equip P1/P2 rules with runbooks, on-call routing, inhibition, maintenance silences; observe noise in shadow mode and calibrate thresholds.

Week 3 – Make Diagnosis Repeatable. Build overview, instance, replication/lock/capacity specialty dashboards; integrate change and CMDB links; version-control dashboards and rules.

Week 4 – Validate Resilience. Drill at least one replica lag, one instance unreachable, one alerting chain failure; record MTTA/MTTR, revise rules, runbooks, and release checks.

Conclusion

Prometheus, Grafana, and Alertmanager are merely implementation means. True "fire prevention" capability comes from verifiable design: using SLIs to connect database state to user impact; using controlled labels and capacity budgets to keep platform sustainable; using recording rules and layered architecture to support scale; using routing, inhibition, runbooks, and drills to make every alert actionable; then feeding every incident's lessons back into rules, code, and release processes.

When the monitoring system can warn of capacity risks before failure, provide clear remediation entry points during failure, and drive systemic improvements after failure, the team has truly transformed from firefighting to fire prevention.