How to Build Scalable Observability for Cloud‑Native Environments: Lessons from SRE

1. Challenges in the Cloud‑Native Era

Enterprises moving from monolithic to SOA architectures and then to microservices and containers face three major challenges: diversity (heterogeneous languages, multi‑cloud, multi‑region, hybrid legacy‑cloud coexistence), dynamism (containerization, rapid deployment and scaling), and massive scale (thousands of services, millions of metrics). These challenges require multi‑cloud, multi‑region, multi‑cluster designs to avoid single‑cloud failures.

2. Observability Architecture

The goal is to reduce fault frequency and mean‑time‑to‑recovery (MTTR) by providing fault perception, localization, and remediation capabilities. Core observability consists of Traces, Logs, Metrics, and Events, collected from diverse resources, then processed, stored, and correlated for application‑centric analysis.

CMDB evolves through four stages: CMDB 1.0 – digital asset inventory; CMDB 2.0 – platform‑wide data modeling and auto‑discovery; CMDB 3.0 – application‑centric view linking services, databases, caches, etc.; CMDB 4.0 – a digital map enabling intelligent operations such as root‑cause analysis.

Metric collection uses decentralized Prometheus instances (≈150) per cloud/region. To avoid single‑point failures, Thanos aggregates data via sidecars, providing a unified query layer while preserving decentralization.

High‑cardinality metrics are mitigated by reducing dimensions and leveraging VictoriaMetrics flow processing (vmagent, vmgateway). This pipeline ingests raw metrics, performs stream‑level aggregation, and writes back to Prometheus, achieving sub‑second query latency for large‑scale data.

3. Problems and Solutions During Construction

Application‑centric CMDB : phased from 1.0 to 4.0, enabling automated resource discovery and relationship building.

Decentralized collection & storage : Prometheus + Thanos for multi‑cloud metric gathering; Solo component discovers resources via CMDB and dispatches appropriate Prometheus instances.

High‑cardinality metric handling : lower cardinality and dimensions; use VictoriaMetrics flow engine to aggregate and rewrite metrics, reducing query time from minutes to milliseconds.

Alerting stack : open API gateway for custom alerts, processor linking alerts to CMDB owners, Feishu notification with smart scheduling, escalation, and convergence (aggregation, time‑window, root‑cause analysis, AI‑assisted filtering).

Application‑centric observability platform : provides multiple views—service‑side, client‑side, instance, interface, topology—to quickly pinpoint fault origins.

SLA/SLO framework : extract key SLIs, compose SLOs, set multi‑9 availability targets, visualize burn‑down charts, and align with business agreements.

Productization governance : systematic planning, development, and operation processes (competitor analysis, roadmap, requirement reviews, training) to improve platform adoption and user satisfaction.

4. Future Outlook

Future work focuses on expanding observation dimensions, adopting eBPF for low‑overhead instrumentation, and shifting from experience‑driven to AI‑driven fault perception using AIOps. Open‑source projects DeepFlow (https://deepflow.io/) and Pixie (https://px.dev/) are highlighted for further exploration.