How to Build Reliable Monitoring for Large‑Scale Systems

Introduction

When incidents occur, teams rely on observability to answer where and why something broke. If the observability stack depends on the very systems that are failing, dashboards disappear and alerts stop, defeating the purpose of monitoring.

Airbnb discovered a circular dependency: its metric‑monitoring pipeline was built on the same shared infrastructure it was supposed to observe, creating a reliability risk at scale.

Hidden Risk: Circular Dependency

Observability must be more reliable than the systems it monitors. In Airbnb’s platform, the shared Kubernetes and service‑mesh infrastructure that hosts production workloads also hosted the monitoring components, leading to a feedback loop where a failure in the infrastructure could cripple the monitoring stack.

Isolating Compute

Airbnb evaluated two extremes: running observability components on the shared production cluster (low maintenance but high coupling) and maintaining a completely separate Kubernetes cluster (full isolation but high operational burden). The chosen compromise was to run observability workloads on dedicated Kubernetes clusters that are managed by the Cloud team but not shared with product services, providing isolation while limiting operational overhead.

Refactoring the Network Layer

Because observability traffic is orders of magnitude higher than business traffic, the existing Istio service mesh could not provide the required isolation and priority. Airbnb built a custom L7 proxy based on Envoy, deployed outside the shared compute layer, to load‑balance and route read/write requests separately from the mesh.

This proxy enables header‑based tenant routing, mapping each service name to a specific backend cluster, and allows independent control over traffic shaping, mirroring, and fine‑grained access control.

Monitoring the Monitoring System

To detect failures in the observability stack itself, Airbnb runs a set of independent Prometheus instances (meta‑monitoring) on isolated Kubernetes nodes across availability zones. Each Prometheus‑Alertmanager pair is placed on distinct infrastructure to avoid shared‑failure domains.

A “dead‑man’s‑switch” continuously emits a stable signal; if the signal stops, CloudWatch alarms trigger, notifying on‑call engineers of issues such as Prometheus downtime, scrape failures, or Alertmanager problems.

Conclusion

Designing reliable monitoring requires anticipating the toughest moments—partial failures, network degradation, and dependency loss. By eliminating circular dependencies, isolating workloads on dedicated clusters, decoupling observability traffic from the service mesh, and adding robust meta‑monitoring with a dead‑man’s‑switch, Airbnb achieved a monitoring system that remains available even when the rest of the platform is under stress. These practices can be adapted by any organization seeking to make its observability stack more resilient than the systems it observes.