Designing an Automated Operations System for Hybrid Cloud Environments

Core Challenges of Hybrid‑Cloud Operations

Network inconsistency : An Alibaba Cloud ECS instance accessing an on‑premise Kafka cluster saw latency jump from 2 ms to 200 ms after a routing change on a provider’s intermediate node, discovered only after two days of troubleshooting.

API inconsistency : Alibaba Cloud uses OpenAPI, AWS uses Boto3‑style SDKs, and the private data center relies on SSH commands. The same “create VM” operation requires different parameters, workflows, and response formats across the three environments.

State inconsistency : Each cloud supplies its own monitoring, logging, and alerting. A machine may appear healthy in the cloud console while business logs in the on‑prem ELK cluster report errors, causing failures to be missed without correlation.

Overall Architecture Design

The solution is organized into four decoupled layers, each depending only on the standard interfaces of the layer below:

Infrastructure access layer – masks cloud‑specific differences.

Unified management layer – builds a single view of all resources.

Automation engine layer – drives workflows and self‑healing.

Operations portal layer – provides human‑machine interaction.

Unified Resource Management Layer

CMDB and Asset Topology

A custom CMDB runs discovery agents (Telegraf for on‑prem, vendor SDKs for clouds) that sync inventories every five minutes. Resources receive a uniform identifier {provider}-{region}-{type}-{id}, e.g., aliyun-cn-hangzhou-ecs-i08x7f3b. Relationships are stored in NebulaGraph, enabling a single graph traversal to answer “which services are affected by a given machine’s failure”.

Infrastructure as Code (IaC)

Pulumi is used instead of Terraform because it allows definitions in TypeScript, Python, or Go. All infrastructure changes are submitted via Git pull‑request; after merge a CI/CD pipeline runs pulumi up. Manual console changes are prohibited.

Observability System Construction

Unified Collection with OpenTelemetry

An OpenTelemetry Collector is deployed in each environment to gather metrics, traces, and logs, performing only protocol conversion and light filtering before sending data via gRPC to a central backend.

Metrics are stored in VictoriaMetrics (≈40 % less memory than Prometheus), traces in Grafana Tempo, and logs in Loki. All three support object‑storage cold‑data offload.

Alert Convergence

Alerts are correlated based on resource topology; alerts belonging to the same fault chain are merged into a single root‑cause alert. In production this reduced alert volume by about 65 % and improved effective response rates.

Automation Orchestration and Self‑Healing

Workflow Orchestration with Temporal

Temporal (open‑source fork of Cadence) is used as the workflow engine because it natively supports long‑running workflows, automatic retries, and state persistence.

Example “scale‑out” workflow: receive request → check quota → invoke IaC to create instance → wait for readiness → register in service discovery → run health check → update load balancer. Each step’s state is persisted, so a Temporal restart resumes from the last successful step.

AIOps – LLM Agent Deployment

A large‑language‑model‑based agent is triggered on alerts, automatically pulling related metrics, logs, and change records, then generating a preliminary root‑cause analysis.

Limitation : the LLM Agent only assists diagnosis; it never executes fixes automatically. In one incident the agent incorrectly suggested restarting a database primary, which would have been disastrous without human confirmation.

Self‑Healing Runbooks

Deterministic failures are codified as runbooks and registered with the self‑healing engine, for example:

Disk usage > 90 % → auto‑clean old logs and temporary files

Pod repeatedly OOMKilled → auto‑increase memory limit and restart

SSL certificate nearing expiry → auto‑renew and hot‑load

Approximately 40 runbooks cover ~70 % of common failures, reducing mean time to recovery from 45 minutes to 8 minutes.

Security, Compliance, and Cost Governance

Zero‑Trust Networking

Cloudflare Zero Trust is used so every cross‑cloud request undergoes identity and device verification, replacing traditional VPNs.

Secret Management

All sensitive data (database passwords, API keys, private certificates) are stored in HashiCorp Vault and accessed via short‑lived tokens; hard‑coding secrets in configs, environment variables, or repositories is forbidden.

Cost Governance

Each cloud’s billing API is integrated, costs are normalized into a unified model, and spend is attributed by business line, project, and environment. Weekly cost reports flag anomalies; one incident uncovered forgotten GPU instances costing >$2,000 per day for two weeks, demonstrating the value of automated cost monitoring.

Landing Recommendations and Pitfalls

Incremental rollout : The first version only unified monitoring collection and basic alert convergence. After three months of stability, IaC, orchestration, and self‑healing were added.

Adapter pattern : Adding a new cloud or on‑prem environment requires implementing a standard adapter with ~30 methods (create instance, query status, fetch metrics, etc.). Development effort is roughly 2–3 person‑weeks per cloud.

Gray‑release automation : New runbooks are tested in a staging environment for two weeks, then gray‑released to non‑critical production services for another week before full rollout. A runbook that worked in staging failed in production because a cleanup script behaved differently on CentOS 7 versus Ubuntu 22.04; the gray phase caught the issue.

Cross‑cloud network reliability : Primary dedicated lines are not 100 % reliable. A backup public‑VPN link with automatic failover within 30 seconds rescued the system three times in a year.

Hybrid‑cloud automated operations require unified abstraction, layered decoupling, and gradual implementation. There is no single silver bullet; the design must fit the team’s stack and business needs, solving the most painful problem first and iterating onward.