10 Essential Grafana Dashboards to Spot Incidents Early

The article lists ten Grafana dashboards that, when built on first‑principles and tuned to surface signal rather than noise, help teams detect incidents early. It assumes metrics are collected with Prometheus/PromQL, logs with Loki, and traces with Tempo/OpenTelemetry, but the patterns can be adapted to other stacks.

01. SLO Pulse – Error‑Budget Burn (Canary in a Canary)

What it tells you: Whether your error‑budget is currently burning.

Panel: 4‑hour and 30‑day burn rates, burn multiplier, rolling success rate, and the endpoints with the most failures.

PromQL (multiple windows, multiple burns):

# 99.9% SLO
ratio_5m = sum(rate(http_request_duration_seconds_count{status!~"5.."}[5m])) / sum(rate(http_request_duration_seconds_count[5m]))
ratio_1h = sum(rate(http_request_duration_seconds_count{status!~"5.."}[1h])) / sum(rate(http_request_duration_seconds_count[1h]))
burn = ((1 - ratio_5m) / (1 - 0.999)) + ((1 - ratio_1h) / (1 - 0.999))

How it works: The burn rate aligns alerts with customer‑impacting errors rather than spurious CPU spikes.

02. User‑Journey Funnel – Where Conversions Drop

What it tells you: Which step in the "search → product → checkout → payment" flow is breaking, near‑real‑time.

Panel: Step conversion percentages, week‑over‑week delta, P95 step latency, and a heatmap of drop‑offs by region.

PromQL (per‑step success ratios):

# Per‑step success ratios
sum(rate(app_event_total{event="checkout_success"}[5m])) / ignoring(event) group_left sum(rate(app_event_total{event="checkout_start"}[5m]))

How it works: You can capture broken buttons, third‑party failures, or feature‑flag issues long before they become visible to users.

03. Infrastructure USE Dashboard (Utilization, Saturation, Errors)

What it tells you: CPU, memory, and I/O pressure together with actual saturation (queues, limits).

Panel: Node CPU utilization, runnable queue length, disk utilization %, disk wait time, network packet loss, and throttled containers.

PromQL:

# CPU saturation (runnable tasks)
node_load1 / count(count(node_cpu_seconds_total{mode="idle"}) by (instance))
# Disk saturation
rate(node_disk_io_time_seconds_total[5m])

How it works: The USE method exposes remaining headroom, showing bottlenecks before a metric hits 100%.

04. Queue & Back‑pressure – "Can We Keep Up?"

What it tells you: Latency and throughput of Kafka/Rabbit/SQS consumer groups and their health.

Panel: Per‑consumer‑group lag, production vs. consumption rate, dead‑letter count, age of the oldest message.

PromQL:

# Kafka consumer lag (exporter varies)
max(kafka_consumergroup_lag{group="payments"}) by (topic, partition)
# Oldest message age (seconds)
max_over_time(kafka_topic_oldest_message_age_seconds[5m])

How it works: Slow consumers are often the earliest symptom of downstream problems.

05. Database Health – Connections, Locks, Slow Queries

What it tells you: Whether Postgres/MySQL are healthy under real load.

Panel: Active vs. max connections, lock‑wait count & duration, P95 query time by verb, top‑N slow queries (via Loki), replication lag.

PromQL (Postgres):

# Waiting locks
sum(pg_locks_count{mode!="AccessShareLock",state="waiting"})
# Replication lag seconds
max(pg_stat_replication_lag_seconds)

Loki (slow‑query table):

{app="postgres"} |= "duration:" | json | duration > 200ms | stats count() by query

How it works: Databases can fail gracefully—lock‑wait spikes signal trouble before a crash.

06. Cache Lie Detector – Contextual Hit‑Rate

What it tells you: Whether you are paying unnecessary main‑store costs.

Panel: Hit/miss rate over time, miss reasons (cold vs. evicted), P95 cache latency, bytes evicted.

PromQL (Redis/Dragonfly):

sum(rate(redis_keyspace_hits_total[5m])) / (sum(rate(redis_keyspace_hits_total[5m])) + sum(rate(redis_keyspace_misses_total[5m])))

How it works: A subtle decline in hit‑rate is an early, low‑cost warning.

07. CDN/Edge Early Warning – Latency, TLS, Origin Errors

What it tells you: Whether the internet itself is the problem.

Panel: P90/P99 edge latency by POP, TLS handshake time, origin 5xx error rate, cache status (HIT/MISS/BYPASS).

LogQL (structured edge logs):

{source="edge"} | json | status >= 500 | stats sum(count) by pop, route

How it works: Quickly distinguishes origin issues from problematic POPs or ISPs, reducing wasted investigation.

08. Rollout Guardrails – Feature‑Flag & Cohort Error Increments

What it tells you: Whether a new feature is impacting a specific cohort.

Panel: Error‑rate delta, P95 latency delta, affected primary endpoints, automated rollback hints.

PromQL (cohort comparison):

err_on = sum(rate(http_requests_total{status=~"5..",flag="on"}[5m])) / sum(rate(http_requests_total{flag="on"}[5m]))
err_off = sum(rate(http_requests_total{status=~"5..",flag="off"}[5m])) / sum(rate(http_requests_total{flag="off"}[5m]))
delta = err_on - err_off

How it works: Provides causal hints rather than mere correlation, ideal for progressive delivery.

09. Trace Topology Mini‑Map – Where Time Is Spent

What it tells you: Which service or endpoint caused the latest latency spike.

Panel: Service dependency graph (Tempo/OTel), P95 span duration by operation, error hotspots, auto‑focused "red nodes".

Grafana conversion: Use the Service Graph panel and a table listing service.operation, p95, error_rate.

How it works: Traces eliminate finger‑pointing; problematic nodes light up red, guiding you to the right team.

10. On‑Call Situation Room – One‑Click Command Center

What it tells you: "Are we OK?" If not, the first place to look.

Layout:

Row 1: SLO burn gauge, active alerts (small table), on‑call contact.

Row 2: P95 latency + error rate (global), request volume, saturation (CPU/queue).

Row 3: Recent 5 deployments (annotations), endpoint‑wise errors, major slow queries.

Row 4: Runbook links and a "silence‑noise" toggle (compare with last week).

How it works: Provides a cockpit view during incidents, avoiding navigation across many dashboards.

11‑13. Operational Tips

Budget‑aware alerts: Configure SLO‑burn alerts at 2× (fast) and 1× (slow) burn, with grouping tags {service, region}. Fast alerts fire for 10 minutes, slow for 30 minutes.

Queues: Alert on lag slope ( deriv(lag[10m]) > 0) instead of absolute lag.

Database: Trigger on lock‑wait > 20 s for > 3 min or replication lag > 60 s.

CDN: Alert when any POP’s origin 5xx error rate exceeds 1 % within 5 minutes.

Additional habits for compounding benefits include time‑shifts on key panels, CI/CD annotations for deployments and feature flags, consistent units (ms, %), and linking each panel to a deeper drill‑down view.