How to Build a Future‑Proof Operations Platform with End‑State Architecture

1. Business Background

Rapid growth of internet traffic has forced infrastructure to evolve while supporting both new and legacy architectures. Different business units have varying operational requirements, leading to many possible operation paths (migration, scaling, serverless, etc.). Choosing the right path is uncertain, making traditional process‑oriented operations unsustainable.

Industry leaders (Microsoft AutoPilot, Google brog, Alibaba Apsara) adopt a common principle: the end‑state model, where users specify the desired final state and the system automatically determines the execution path based on real‑time conditions and knowledge bases.

2. Architecture Overview

The system consists of several components:

Portal : Business‑oriented form for users to submit change requests.

API Server : Validates requests, performs static security checks, rate limiting, approval, and translates them into an end‑state SPEC.

Brain : Splits the end‑state SPEC into ordered SubSPECs according to operational safety rules.

Master : Stores desired and current states of all operational instances.

Controller : Reads desired states from Master and drives the system toward the end‑state.

Decider : During state convergence, decides whether an operation can proceed based on the current environment.

Monitor : Continuously reports the actual state of each instance.

The workflow mirrors a PDCA loop: controller proposes actions, decider approves based on global health, controller executes, monitor reports, and the system iterates until the desired state is reached.

3. Safety Model

Safety depends on two key factors:

Topology of operational instances – understanding the impact scope of a change.

Application profile – health indicators such as version, latency, traffic, etc., beyond simple version checks.

These factors guide the Decider to allow or reject actions, ensuring reliable operations even in complex, multi‑region, multi‑cloud environments.

4. Real‑World Problems

Design choices must balance system complexity and team capacity. A Master‑Controller‑Agent model distributes load, allowing Controllers to batch operations while Agents handle execution on diverse OSes.

Stateful scenarios (e.g., targeted scaling, ordered group deployments, machine‑specific replacements) are addressed by allowing operational parameters that influence runtime behavior, bridging the gap between stateless descriptions and stateful execution.

CMDB serves as the authoritative source of operational data; its reliability directly affects global planning and disaster recovery.

5. Disaster Recovery Strategies

Two typical approaches for database resilience:

Unit‑level deployment across regions for horizontal scaling, at the cost of higher complexity.

Primary‑secondary replication with failover, simpler but may overload the backup during failures.

The chosen hybrid strategy applies strong consistency to critical data and eventual consistency to less critical data, balancing performance and reliability.

6. Future Outlook

Exploration of AIOps aims to formalize operational rules (e.g., deployment constraints) and compute optimal execution paths, though practical adoption remains limited due to data quality, standardization, and experience requirements.

Knowledge‑base‑driven frameworks, like the end‑state model, often provide more immediate value than deep learning approaches.

Overall, the end‑state architecture offers a simple logical model that can be complex to implement, requiring careful handling of stateful scenarios, multi‑cloud integration, and operational safety.