How to Build a Data‑Driven Stability Assurance System for Kubernetes Clusters

The "Kubernetes Stability Assurance Handbook" series introduces a comprehensive, systematic method for ensuring cluster stability by modeling complexity, digitising insights, and visualising actions.

Sources of Stability Complexity

Complexity arises from multiple dimensions:

Number and interaction of system components – continuously evolving.

Dynamic behavior of components and interactions – hard to infer and observe.

Types and quantities of system resources – also change over time.

Dynamic behavior of resources – similarly opaque.

Stability‑ensuring actions – difficult to standardise and execute safely.

Addressing these requires effective, comprehensive insight into the cluster and safe, pre‑approved action plans.

Data Model Overview

The model is expressed through four diagrams and three tables that abstract insight and pre‑plan information.

Four Diagrams

Architecture Relationship Diagram : describes components and their static interactions.

Architecture Runtime Diagram : captures dynamic characteristics of components and interactions.

Resource Composition Diagram : shows how resources are assembled.

Resource Runtime Diagram : depicts dynamic usage characteristics of resources.

Three Tables

Event List : events that require attention.

Action List : approved management actions.

Plan List : mappings from events to actions (pre‑plans).

Insight Layer

Insight focuses on two core aspects: cluster architecture and cluster resources.

1. Architecture Relationship Diagram

Components are represented as nodes and interactions as edges. Example JSON representation:

{
    "nodes": [
        {"_id": "0ce0e913f6e5516846c654dbd81db6ecab1f684e", "name": "kube-apiserver", "type": "managed component", "dependencies": {}},
        {"_id": "f0740d8bb67520857061a9b71d4a9e4fc50bfe3d", "name": "etcd", "type": "managed component | storage", "dependencies": {}},
        {"_id": "05952a825e91cb50a81cbaf23c6941d5c3bb2c89", "name": "eni-operator", "type": "component", "dependencies": {"serviceaccount": "enioperator", "clusterrole": "enioperator", "clusterrolebinding": "enioperator", "configmaps": ["eniconfig"], "secrets": ["enioperator"]}}
    ],
    "edges": [
        {"_id": "38bce9ca8a0cec6d8586d96298bd63b0523fc946", "source": "eni-operator", "target": "kube-apiserver", "description": "manage ENI requests"},
        {"_id": "93f3c21247165f0be3a969fc80f72bc1a402e9f5", "source": "eni-operator", "target": "Network Service", "description": "manage VPC/VSwitch via Alibaba Cloud API"}
    ]
}

2. Architecture Runtime Diagram

Runtime data (logs, metrics, traces) is overlaid on the static architecture to visualise health. Example JSON snippet shows nodes with insight sources and signal definitions.

{
    "nodes": [{
        "_id": "ea4538dc0625d06b0dc93579998e04288656050f",
        "name": "mutatehook",
        "deploy": {"type": "K8s:Deployment", "namespace": "kube-system", "replicas": 3},
        "insight": [{
            "source": {"vendor": "cloud:aliyun:sls", "log_project": "xxx", "log_store": "mutatehook", "log_url": "https://sls.console.aliyun.com/lognext/project/xxx"},
            "signal": {"exception": {"fuzzy": "fail OR Fail OR error OR Error"}}
        }]
    }],
    "edges": [{
        "_id": "38bce9ca8a0cec6d8586d96298bd63b0523fc946",
        "source": "eni-operator",
        "target": "kube-apiserver",
        "insight": [{"source": {"vendor": "cloud:aliyun:sls", "log_project": "xxx", "log_store": "xxx", "log_url": "https://sls.console.aliyun.com/lognext/project/xxx"}, "signal": {"exception": {"unauthorized": "Unauthorized", "throttling": "'Throttling' OR 'throttling'"}}}]
    }]
}

Resource Layer

Resources are modelled as a graph where nodes are resources and edges represent ownership or binding.

{
    "kinds": ["vpc","vswitch","securitygroup","ecs","clb","rds","nat","eip"],
    "tags": {"cluster/product": "xxx", "cluster/id": "2736f42d4e882ad6825d6364545a3f1cb5136859", "cluster/name": "xxx", "cluster/env": "staging"},
    "nodes": [{"kind": "vpc", "nodes": [{"_id": "c505f21871bac7385c1387988cf226310af0831e", "id": "vpc-xxx", "ipv4": "xxx", "tags": {"resource/creator": "product"}, "url": "https://vpc.console.aliyun.com/vpc/xxx"}]},
    {"kind": "ecs", "nodes": [{"_id": "47c4fe5cc2585a49f07798a0b8b69cda7f8d4a23", "id": "xxx", "az": "xxx", "interfaces": {"primary": {"ip": "xxx", "eni": "xxx", "mac": "xxx"}}, "instance-type-family": "xxx", "instance-type": "xxx", "tags": {"resource/creator": "product", "resource/role": "worker"}, "url": "https://ecs.console.aliyun.com/#/server/xxx"}] }],
    "edges": [{"_id": "a754c748b2723a25c017421dd0969d00df3c000b", "source": "vsw-xxx", "target": "vpc-xxx"}, {"_id": "c34b164eba2897cfb2b574a576672d8aa441d709", "source": "eip-xxx", "target": "ngw-xxx"}]
}

Pre‑Plan Layer

Events, actions, and plans are defined using the CloudEvents specification for events.

{
    "events": [{"_id": "a1ab5b61857be35a5c5b203dd84b49248161c823", "description": "restart workload manually", "event": {"id": "restart-workload", "source": "xxx", "specversion": "1.0", "type": "com.aliyun.trigger.manual", "datacontenttype": "application/json", "data": "{\"NAMESPACE\": \"\", \"NAME\": \"\", \"TYPE\": \"\"}"}}]
}

{
    "actions": [{"_id": "47abc5cd9d64018ebf96dc5b2d6a4fbd35a3cb6d", "name": "Action Restart Workload", "exec": "restart-workload", "env": ["NAMESPACE","NAME","TYPE"]}]}

{
    "plans": [{"_id": "29a091c48d8992991ed69e8694b017a11abe3eec", "name": "Plan Restart Workload", "description": "重启 workload", "event": "a1ab5b61857be35a5c5b203dd84b49248161c823", "actions": ["47abc5cd9d64018ebf96dc5b2d6a4fbd35a3cb6d"]}]}

Global Visualisation Service

Combining the four diagrams and three tables yields a core "insight + pre‑plan" model that can be exposed as a globally visualised stability‑assurance service. Key attributes are a global perspective, digitisation, and visualisation.

Implementation Sketch

Deployment shape : region‑level deployment serving one or multiple clusters per region.

Usage experience : organise stability practices into columns such as Run‑Chain Diagram, Deployment Architecture, Business Flow, Data Analysis, Observability Management, and Controllability Management.

Operational Workflow (Normal State)

Use the Data Analysis column to verify coverage and precision of observability and controllability.

Manage observability dimensions (data sources, monitoring, alerts, governance) in the Observability Management column.

Configure pre‑plans, issue management, and chaos‑engineering results in the Controllability Management column.

Overlay configured monitoring, alerts, and pre‑plans onto Run‑Chain and Deployment diagrams for visual guidance.

Incident Handling Workflow

Detect anomalies via run‑chain diagrams or alerts.

Trigger issue tracking automatically or manually.

Identify affected components and severity through colour‑coded graph elements.

Drill‑down on anomaly numbers to view detailed event data or jump to log/tracing systems.

Select the appropriate pre‑plan based on the issue.

Execute the pre‑plan directly from the run‑chain diagram (block or recover services).

Confirm execution effect via updated visual cues.

Close the issue and record a snapshot of the run‑chain diagram.

Issue Tracking Details

Issue identifier

Timestamp of anomaly occurrence

Actions performed during handling

Run‑chain diagram snapshot

Timestamp of recovery

Data Model Competitiveness

The data model serves as a medium for iterating, sharing, and applying best‑practice stability assurance. Generic insight + pre‑plan services can be standardised, while custom ones are described with the same structure and realised via a common controller.

Insight Model : answers "How to observe cluster stability?" and "How to gauge business iteration efficiency?"

Data Model : addresses "How to define an effective, extensible data description?"

Competitive advantages focus on global, digital, visual insight; operational efficiency (shortest action path, minimal cost); and advanced, process‑driven best practices.

Conclusion

By specifying seven data‑model schemas, the "insight + pre‑plan" approach provides a structured, repeatable foundation for stability assurance, accelerates business iteration, and can even feed back into product direction.