Understanding Monitoring Systems: MTTF/MTTR, Three Signals, and AI Observability

1. The Fundamental Question: Why Monitor?

Without monitoring, your customers become the ones answering: Is the app running? Is it fast enough for everyone? Will it crash in the next hour? When it fails, which component broke and how quickly can you find out? Availability is defined as MTTF / (MTTF + MTTR), where MTTF is mean time to failure and MTTR is mean time to recovery. Monitoring extends MTTF by detecting degradation before failure, and reduces MTTR by making you the first to know, enabling faster diagnosis and verification of fixes. The book Accelerate states that proactive monitoring correlates strongly with both performance and job satisfaction.

2. Three Telemetry Signals

Collectable data is called telemetry:

Metrics : Numeric measurements; storage, transmission, and query are extremely efficient, but granularity is coarse and they cannot drill down. They only reveal overall system health and whether something is broken.

Logs : Text output from the system; they supplement metrics with detail but can easily fill disks.

Traces : In distributed microservice architectures, single-service logs cannot reconstruct the full picture.

OpenTelemetry standardizes collection and export; it is not a backend. Data must still be sent to a storage system. Its value is vendor/tool portability without rewriting application instrumentation. Instrumentation should emit data outward — not keep it locally — because you need to observe trends over time and retain queryability even when the application itself is down.

3. Metrics: Four Golden Signals and a Cardinality Trap

Google SRE defines four golden signals:

Latency : Request duration. Rising latency indicates increasing load or degradation; investigate before it crosses error thresholds.

Traffic : Request volume. Distinguishes normal growth from attacks.

Errors : Failure rate of incoming requests.

Saturation : Distance to resource exhaustion.

Metrics backends store time series; each unique label combination creates a series. Low-cardinality labels (host, API, status code) are safe. High-cardinality labels (username, email, client IP) cause series explosion.

4. Logs: Separate Two Types of Errors

Logging every error or exception as ERROR is wrong. User input validation failures are user errors, not server errors; log them as WARN or INFO. This keeps error rate reflecting server health, not user mistakes. Retry logic should log first failure as WARN, escalate to ERROR only after retries exhaust. Structured logging (following OTel semantic conventions) makes querying, aggregation, and alerting far easier. Avoid free-text logs that cannot be reliably parsed. Logs are a major cost driver; excessive volume or retention burns money.

5. Tracing: One Request, Four Hops

With an OTel agent, a full trace captures the request flow:

GET /joke                          server span (starts earliest, ends latest, reported last)
├── GET api.chucknorris.io         client span, 200, 143ms
├── SELECT ...                     db span, internal
└── INSERT INTO jokes ...          client span, db

Component usage and per-call error rates are visible at a glance; drill into any trace for details.

6. Sampling and an Alternative Path

High volume requires sampling, but sampling must be per whole trace — partial traces are meaningless. Two common strategies:

Head sampling : Decide at request start. Simple, but you don't know the outcome; a rare failure may be discarded.

Tail sampling : Decide after request completes. Keep failures and slow requests; drop normal ones. Accurate but requires buffering the entire trace.

A different approach is the wide event model (advocated by Honeycomb, Axiom): pack all request attributes into a single event written to a columnar database. Logs, traces, and metrics can be derived from it. The rationale: you cannot know in advance what you will need to query, so avoid premature trade-offs.

7. Visualizing Data

Dashboards must not share infrastructure with application servers; if the app crashes, you lose visibility into what just happened.

Multiple simple dashboards beat a single all-encompassing wall. Only daily users know which panels matter; newcomers face an information wall.

Annotate deployments on the timeline. A sudden metric shift after a release is often the starting point for investigation.

Grafana is the best open-source solution, excelling at merging disparate data sources.

8. Defining Alerts

Thresholds that warrant waking someone differ across environments (test vs. production traffic). Conditions must persist long enough to avoid alerting on transient spikes.

Treat alert rules as code: version-controlled, rollbackable, environment-specific, tested in staging before production.

Alert on symptoms, not causes. Root causes are infinite and unpredictable; ultimate symptoms are few (service unresponsive, requests failing). Symptom-based alerts cover broad failure classes.

Avoid alert fatigue. Hundreds of daily alerts breed habituation; a critical certificate-expiry alert gets ignored.

Alerts must be actionable. The recipient should know the next step from the alert itself, not by searching source code for the alert's author.

9. Three Advanced Practices

Synthetic testing : Continuously simulate real users in production — hit the same APIs or run headless browser flows. Alert on deviations. Value is timing: catch bugs on deploy, not weeks later after multiple releases obscure the culprit. For payment systems, use test accounts that downstream discards, or accept a penny of real transactions daily for a definitive answer.

Real User Monitoring (RUM) : Extend monitoring into browsers and apps. Some errors are only visible to end users: broken CSS (white-on-white), client download failures (backend only sees traffic drop), or backend latency so low it masks bad API design (N+1 problems emerge over real networks).

Prediction : Disk not full now, but in four hours? Example PromQL:

predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600) < 0

Extrapolate the last hour's consumption rate four hours forward. Observability Engineering suggests looking back X time to forecast ~4X future. When normal baselines are unknown, use ML-based anomaly detection to draw a historical band.

10. Organizing On-Call

Tools like PagerDuty or Jira Service Management manage rotations, escalations (5 min no response → next teammate → manager). Manual handling is tedious; tools are worthwhile. More important than tools is drilling. Teams with rare incidents often fumble at 4 AM: unsure where to look, expired permissions, stale passwords, broken PIM grants. Regular drills expose these gaps in daylight. After incident resolution, conduct blameless postmortems: what should have alerted but didn't? Which noisy alerts drowned the critical one? Periodically review the entire monitoring configuration — does it still match the current system?

11. AI in the Observability Loop

AI can occupy five levels during incidents:

L1 Q&A : Paste stack traces/logs to a chatbot; ask what commands to run or metrics to check. No system permissions; safest but copy-paste overhead. Guard against leaking sensitive data.

L2 Local Read-Only Diagnosis : Agent uses CLI tools or MCP servers to inspect system state, iterates, proposes diagnosis and fix. You execute. Prerequisite: provide context — components and how to check each. The agent has seen more incidents than any engineer but doesn't know your system.

L3 Hands-On Repair : Restart, rollback, commit fixes — you approve each action. Boundaries must be explicit: allow restart? allow DROP DATABASE?

L4 Remote Async : Agent lives in the monitoring system; on alert it starts investigating while you sleep. You wake to a written report. Still read-only; saves the least productive 20 minutes after being paged.

L5 Full Automation : It detects, fixes, verifies; you keep sleeping.

L5's challenge is cultural, not technical. Postmortem culture requires human accountability. If an agent silently fixes a 4 AM incident, no alert fires, no postmortem occurs, the failure is quietly buried. Removing the human leaves responsibility nowhere.