How ACK Automates Fault Self-Healing Across Heterogeneous GPU Clusters at Scale

Background: Stability Challenges in Large-Scale Heterogeneous Training

As training clusters scale from hundreds to tens of thousands of GPUs, hardware failures become a constant rather than an exception. Distributed training's strong synchronization means a single GPU or node failure stalls the entire job at communication barriers, unlike stateless online services where traffic simply reroutes. This amplifies single-point failures into global stalls, making stability a first-class concern. The cost of failures grows super-linearly with cluster size: mean time between failures drops while the cost per failure rises due to larger job footprints. Traditional manual operations face three bottlenecks: delayed detection and diagnosis (signals buried in kernel logs and GPU error codes requiring scarce expertise), long and error-prone remediation chains (isolation, drain, repair, validation, re-admission), and inconsistent handling quality across operators and shifts.

The article introduces effective compute throughput as the north-star metric: the fraction of total cluster compute that translates into actual training progress. Failures erode this metric along three dimensions: detection/response latency (compute spins wastefully while faults go unnoticed), resource idleness during repair (nodes sit offline for hours), and progress rollback (interruptions without checkpointing discard thousands of GPU-hours). The goal of auto-healing is to systematically compress losses across all three dimensions.

System Overview: A Five-Layer Closed Loop

ACK's fault auto-healing system codifies the complete mental model of senior engineers—detect, diagnose, notify, decide, isolate, repair, validate, un-isolate—into an automated pipeline with human guardrails. The architecture comprises five layers, each with a single responsibility, communicating via standard Kubernetes interfaces:

Operations Event Layer: Raw events from heterogeneous infrastructure—Lingjun bare metal (CMS events), EGS, and general ECS (system events).

Detection & Conversion Layer: Node Problem Detector (NPD) runs as a DaemonSet performing fine-grained hardware/software diagnostics; an event converter translates upstream operations events into Kubernetes-native semantics (Node Conditions, Kubernetes Events, Custom Resources).

Notification Layer: All anomalies surface as Node Conditions and Kubernetes Events, fed into the alerting center (DingTalk, SMS, etc.).

Decision & Orchestration Layer: The auto-repair controller binds node faults to user-defined repair workflows per node pool, generating and driving execution plans.

Execution Layer: A set of executors carries out each step—quarantine, drain, repair, validation, un-quarantine—calling underlying Alibaba Cloud IaaS APIs (ECS/Lingjun) for actual remediation.

Layering allows independent evolution: new detection items, new infrastructure integrations, or revised orchestration strategies can be added without global rewrites.

Three Core Design Principles

Decouple detection, decision, and execution. Detection only reports; decision only orchestrates; execution only acts. They communicate via standard status objects. This eliminates human handoffs, slashing detection-to-response latency.

Automation with controllability. The system defaults to automatic progression but returns ultimate control—"whether and when to touch my nodes"—to users. Each repair step can independently require approval, implemented via Kubernetes node labels (e.g.,

intervention.autorepair.node.k8s.aliyun.com/quarantine-node=inquiring

→ approved). High-risk actions wait until workloads have checkpointed, suppressing progress rollback.

Full observability with timeout fallbacks. Every state transition emits traceable events; every wait has a timeout that escalates to alerts and manual takeover. This prevents automation from "running away" on abnormal paths and keeps repair-time resource idleness visible and bounded.

Underpinning these is the "Kubernetes as the operating interface" principle: regardless of underlying platform (Lingjun, ECS, EGS), faults are expressed as Node Conditions, approvals via node labels, and progress as Kubernetes Events. No proprietary control plane is introduced; the system "grows on Kubernetes" using only native objects. This lets operators, upper-layer Operators, and training platforms interact with auto-healing using familiar declarative Kubernetes semantics, while heterogeneous complexity is absorbed beneath the interface.

Fine-Grained Detection: From Raw Signals to Structured Diagnostics

Auto-healing starts with detection. Heterogeneous faults manifest as kernel log error codes, device counter jumps, etc.—invisible to conventional resource monitoring. ACK's NPD runs on each node as a DaemonSet, diagnosing GPU, NVSwitch, NVLink, VRAM, drivers, and critical services.

Heterogeneous nodes have a vastly larger fault surface than generic servers: GPU chip errors, VRAM ECC/row remapping anomalies, NVLink/NVSwitch link failures, PCIe lane degradation, driver/kernel version mismatches, management service outages, and even "soft faults" like GPU resource accounting mismatches on the scheduling side. Each has unique signal shapes and remediation paths. NPD converges these opaque, expert-dependent signals into structured, actionable diagnoses—each tagged with severity, whether device isolation is needed, and a suggested repair direction (reboot, hardware repair, enable ECC, etc.). This turns "detect and understand" from a personal skill into a platform capability.

Example: An XID 79 error appears in Node Conditions and Kubernetes Events with structured fields (type, reason, message). The system also supports GPU card-level isolation : when a fault warrants isolation, NPD emits a faulty-device list; the ACK Device Plugin removes the corresponding GPU from schedulable resources, achieving rapid containment. Isolation is opt-in per detection item and does not equal repair—the node continues to be billed, and the hardware issue persists. Card-level isolation must pair with node-level auto-healing to truly stop loss and fix hardware, forming the basis of "cascaded auto-healing" described later.

Deep Collaboration with Underlying Infrastructure: ECS / EGS / Lingjun

Beyond detection, a critical yet often overlooked step is converting infrastructure operations events into Kubernetes-standard semantics. Infrastructure events are stateful and transition through phases (inquiry → execution → completion); Kubernetes node state is declarative, describing current fact. The converter must reliably map a flowing event state machine into a declarative node status, handling deduplication, independent closure for different event sources/resource granularities (regular nodes vs. super nodes), and precise recognition of terminal states so upstream can promptly judge "fault cleared." Encapsulating these details in the conversion layer lets upstream consumers receive clean, stable, unambiguous node state—decoupling in practice.

Choosing declarative node state as the common language ensures idempotence and self-healing at the system level: regardless of intermediate retries, retries, or controller restarts, as long as the current node state is clear, the decision is consistent. This current-state-based design is a reliability cornerstone for production systems that run at scale for long periods with frequent component restarts.

4.1 Lingjun CMS Events

Lingjun bare-metal hardware/system anomalies are published as CMS events with lifecycle Inquiring → Executing → Completed/Terminated. ACK periodically pulls these events, identifies types (e.g., Node.HardwareFault.HOST for repair, Node.FaultNeedReboot.HOST for reboot), and maps them to corresponding Node Conditions.

4.2 ECS/EGS: System Events & Active Probing

ECS/EGS nodes reuse the ECS System Events framework (states: Inquiring, Pending, Executing, Completed, Avoided, Canceled, Failed). ACK calls ECS APIs to pull instance operations events (e.g., SystemMaintenance.Reboot, SystemMaintenance.Redeploy) and maps them to node state. Additionally, for EGS scenarios, ACK builds in active deep probing : when a managed node pool's NPD detects specific GPU faults, ACK proactively invokes ECS resource diagnostics for deep GPU diagnosis. If ECS system event trigger conditions are met, ECS emits the corresponding system event, which ACK converts to Node Conditions and K8s Events. This shifts stability governance from "wait for event reporting" to "proactive root-cause discovery," catching progressive hardware degradation before catastrophic failure.

4.3 GPU Instance Fine-Grained Detection & Card-Level Isolation

EGS/Lingjun nodes host massive inference and training workloads. ACK combines ECS system events / Lingjun CMS event conversion with NPD heterogeneous detection and card-level isolation: users can enable GPU card-level isolation on demand—when severe card faults are detected, the bad GPU is immediately removed from allocatable resources, then node-level auto-healing handles subsequent remediation. For multi-GPU inference scenarios, this "one bad GPU doesn't affect the rest" granularity is critical.

All three infrastructure types, despite differing underlying mechanisms, converge via the event conversion layer into unified node state semantics consumed by a single decision and execution engine. This is the value of "event source and semantic decoupling": underlying platforms can differ and evolve independently, while upstream logic always faces a uniform node state.

Orchestrated Repair Workflows: Custom Strategies per Fault Class

After identification, how to repair? ACK's principle: no single prescribed path; instead, allow a tailored remediation playbook for each fault class. A repair plan consists of sequentially orchestrated steps; a typical full flow includes four phases— Quarantine (block new scheduling), Drain (gracefully evict existing workloads), Repair (reboot, hardware repair via IaaS, or service restart), Un-quarantine (validate then restore scheduling). But the full flow is merely a default, not a mandate: minor software faults may need only "Repair"; critical online workloads may want "Quarantine only, no drain yet"; some scenarios need only "Quarantine + alert." Each fault class gets its own on-demand playbook.

A table illustrates three example faults with divergent strategies:

GPU severe hardware error: Quarantine (approval) → Drain → Repair (hardware repair) → Un-quarantine.

GPU recoverable error: Quarantine → Drain → Repair (reboot) → Un-quarantine.

Kubelet unhealthy: Quarantine → Drain → Repair (component restart) → Un-quarantine.

Flexibility stems from a two-layer configuration: the upper layer is user-defined auto-repair rules (macro orchestration: which steps, order, approval gates); the lower layer is built-in remediation knowledge (micro details: for a given fault, should "Repair" mean reboot or hardware repair? how long to wait before acting? validation timeout? priority when multiple faults coexist?). The effect: users express intent; the system supplies professional judgment. Even a simple "Repair" selection triggers the correct underlying action—hardware repair for severe errors, reboot for recoverable ones.

When multiple faults coexist on a node, the system arbitrates by remediation priority (Node fault > GPU fault > System/Component fault), tackling the most severe root cause first—avoiding wasteful software restarts on already broken hardware. This "focus on the principal contradiction" saves compute.

The design embodies declarative thinking : users describe "desired remediation strategy," not "step-by-step operations." Declarative brings determinism and reusability—the same policy applies to thousands of nodes and tens of thousands of faults, each executed identically, without drift from scale or personnel turnover. For long-term, large-scale cluster operations, this reproducible consistency is itself a stability source.

ACK categorizes faults entering the auto-healing flow into three classes: Node faults (infrastructure system events), GPU faults (NPD-detected XID errors, reboot-type only), System/Component faults (Kubernetes system component issues). Priority order: Node fault > GPU fault > System/Component fault. Example: if a Lingjun node simultaneously has Node.HardwareFault.HOST CMS event and "container runtime unavailable" component fault, the hardware fault is handled first.

Tiered Authorization: Label-Based Controllable Collaboration

The biggest risk of full automation is acting on nodes running critical workloads. Any quarantine, drain, or reboot/repair may interrupt a days-long training job. Therefore, "whether to act and when" must stay with the business owner. ACK provides step-granular tiered authorization built on the most fundamental Kubernetes mechanism— node labels .

When a repair workflow reaches a step requiring approval, ACK applies an "Inquiring" label to the target node (e.g.,

intervention.autorepair.node.k8s.aliyun.com/quarantine-node=inquiring

), signaling "I'm about to quarantine this node, please confirm," then waits. Only when the label value is changed to "Approved" ( ...=approved) does the workflow proceed. The label key encodes the step name—each approval-gated phase uses its own independent label, configurable per step, realizing per-step authorization mechanically. During the wait, ACK periodically pushes reminders with confirmation instructions; upon detecting the approval label, it immediately continues. Steps without approval requirements proceed automatically. The entire "inquire-approve" handshake is a clean state transition around a pair of node labels.

Using node labels as the authorization vehicle is a seemingly simple but pivotal design. Labeling is the most native, universal Kubernetes operation—any entity with cluster API access (human operators, Kubernetes Operators, training platform controllers) can read "inquiring" labels and write "approved" labels in exactly the same way. Thus ACK's auto-healing system exposes no proprietary interfaces or SDKs for integration; it codifies authorization as a standard, declarative Kubernetes protocol, naturally open to upper-layer business platforms. Existing customer runbooks, approval workflows, custom schedulers, and training frameworks can plug into this authorization loop at near-zero cost—watch labels, decide, write back labels, that's it.

Two key properties:

Authorization is per-step and independent: configurable as "quarantine needs approval, drain auto, repair needs approval, un-quarantine auto," avoiding "one approval, total loss of control."

Graceful exit if node self-recovers during wait: if the fault disappears while awaiting approval, the system skips the step rather than insisting on quarantine/reboot—minimizing business disruption.

Example timeline for a Lingjun Node.FaultNeedReboot.HOST CMS event shows the interplay: CMS event → Node Condition → auto-repair rule match → quarantine (inquiring label) → user/training platform approves → quarantine executed → drain → repair (reboot) → validation → un-quarantine. If the user manually reboots the node during the wait, the system detects recovery and gracefully exits the flow. Customer control over the auto-healing process spans two orthogonal, freely combinable dimensions: Phase selection: Quarantine, Drain, Repair, Un-quarantine can each be toggled on/off, yielding tailored paths (e.g., full loop with approval on quarantine; full loop fully automatic; repair-only for soft faults; quarantine+repair with user-managed drain; quarantine+repair with user-controlled un-quarantine). Per-phase approval toggle: On enabled phases, each can independently require the inquire-approve label handshake. Combined, they cover the spectrum from "fully managed" to "deeply customized." One extreme: no approvals anywhere—fully automatic, fastest, but requires workloads to tolerate arbitrary interruptions (e.g., mature checkpointing and elastic recovery). Other extreme: approvals on every phase—maximum control. Most scenarios land in between, tuned to the business's interruption tolerance. Tiered authorization redefines the automation-human relationship. Traditional thinking treats automation and human control as a binary trade-off: full auto loses control; full manual loses efficiency. Tiered authorization offers a third path: let automation shoulder the vast majority of deterministic, standardizable work, while surfacing the few genuinely high-risk decisions that need human judgment in a clear, timely, programmable way. Automation doesn't "replace humans"; it "amplifies humans" by focusing scarce expert attention exactly where judgment is needed. Integration with training platforms unlocks the full value: when ACK is about to execute a business-interrupting repair step, it posts the "inquiring" label; the training platform's Operator watches for it, triggers a model checkpoint and data backup, then writes "approved"; ACK takes over repair. Zero proprietary integration—the training platform just does what it already does: watch Kubernetes objects and react. A potential "hard interrupt" costing hours of progress becomes an "orderly pause" recoverable from the latest checkpoint, compressing progress loss within one checkpoint interval. This is tiered authorization's most direct payoff: it turns "approval wait time" from pure overhead into a checkpoint window, fundamentally suppressing failure-induced rollback. At large-scale, high-frequency training, the accumulated effective compute savings are substantial. Checkpoint efficiency and elastic training complement this: faster, more frequent checkpoints shrink the rollback window; elastic training lets jobs shrink/expand instead of failing when node count changes. Auto-healing operates at the infrastructure layer (quickly remove, fix, return bad nodes); checkpointing and elasticity operate at the job layer (minimize cost per interruption). Tiered authorization is the key interface linking these layers—it gives the job layer a precise, programmable intervention moment, turning "infrastructure is about to act" into a collaborative event the job can gracefully handle, not a surprise interruption.

Robust Execution: Engineering Guarantees from Quarantine to Validation

Decisions determine "what"; execution determines "whether it goes wrong." If the repair process itself isn't rigorous, automation becomes a new failure source. ACK's execution layer safeguards every phase with "do no harm to workloads" as the prerequisite.

Quarantine & Un-quarantine respect original intent: if a node was manually marked unschedulable before auto-healing started, the system won't automatically make it schedulable again after repair—auto-healing only reverts changes it introduced.

Drain is the most delicate step: uses standard Kubernetes eviction, respects PodDisruptionBudgets, avoids breaching declared minimum available replicas; system-level and explicitly exempted critical workloads are protected; each workload gets a reasonable graceful termination period. A global drain timeout (user-configurable) prevents a single stubborn workload from indefinitely blocking a node that urgently needs repair.

Repair actions map to fault types via underlying capabilities: Lingjun approval-based repair/reboot/spare replacement, ECS instance reboot and system event handling, in-node component restarts for software faults. The execution layer handles unified task dispatch, async tracking, retries, and timeout control.

Validation is the final gate: after each repair step, the system repeatedly re-checks whether the node has truly recovered, until confirmed or the fault-type-specific wait ceiling is hit (hardware repair longest, software faults shortest). Before validation passes, the node is not re-admitted; timeout triggers alert and manual takeover.

These seemingly meticulous guards share one goal: ensure auto-healing doesn't backfire on workloads. Every careful boundary on eviction, graceful termination, validation, and timeout fallback prevents "auto-healing" from becoming a new source of compute waste. The underlying recognition: once an automation system gains the power to modify production nodes, every default behavior must be conservative and predictable. Better to pause and escalate on uncertainty than to take irreversible actions in automation's name. This baseline gives platform teams the confidence to entrust daily operations of tens of thousands of nodes to automation.

Rate Limiting & Circuit Breaking: Safety Valves for 10K-GPU Clusters

In ultra-large heterogeneous clusters, automated repair itself can amplify risk. Burst hardware failures can trigger uncontrolled concurrent repairs causing resource thrashing or cascading failures; systemic risks like datacenter power loss can trap the cluster in a futile auto-retry loop. ACK adds rate limiting and circuit breaking to give automation "safety awareness," keeping every repair within controlled boundaries.

Concurrency rate limiting: configurable max parallel repair nodes—fixed serial count (e.g., 1 at a time) or dynamic percentage of cluster size. Turns a "repair storm" into smooth, ordered processing, smoothing resource volatility and eliminating avalanche risk from mass reboots, preserving running training jobs and stable compute output.

Global circuit breaker: a cluster "fuse." When faulty node count exceeds a preset threshold, the system immediately halts all new auto-repair actions. In extreme scenarios (network reconfiguration, widespread driver crashes), this forces a stop, preventing automation from escalating into a catastrophic accident, buying time for human intervention.

Defensive design with logical interlock: the circuit breaker threshold must be ≥ max parallel repair nodes. This simple rule builds a self-consistent guardrail between rate limiting and circuit breaking, preventing misconfigurations that could cause premature circuit breaking or continued repairs past the breaker threshold.

The "rate limit + circuit breaker + interlock" combo moves stability governance from a pure "speed race" back to "safety and efficiency in balance." It not only recovers lost effective compute but also defends the safety floor of massive AI clusters.

Observability & Alerting: Transparent, Traceable Auto-Healing

Automation must never be a black box. ACK's auto-healing system makes the entire process fully observable with proactive alerts at key nodes.

System-facing: every key action—quarantine, drain, awaiting approval, repair, validation, completion/failure—emits a standardized cluster event, collectable by any observability tool, forming the most detailed timeline for troubleshooting.

Human-facing: auto-healing start, success, failure, and critical wait states (awaiting approval, awaiting recovery) are pushed via the alerting center (DingTalk, SMS, etc.) with context (cluster, node, fault cause). Customers are required to enable ACK event alerting to ensure timely awareness and intervention.

Operators have two clear ways to track a node's auto-healing progress: Kubernetes Events on the node: kubectl describe node <name> shows the chronological event stream—quarantine, drain, awaiting approval, repair, validation, done/failed—making the current phase, approval waits, and errors immediately visible. Auto-healing alert management: enable three core alert rule sets (Cluster GPU Monitoring, Node Pool Operations Events, Node Auto-Healing) with notification channels (DingTalk, Feishu, WeCom, email). Four key alert categories: underlying event conversion alerts (instant awareness of Lingjun CMS/ECS events), GPU monitoring alerts (NPD-detected device faults), auto-healing progress alerts (real-time phase reporting), and daily node pool auto-healing digests (10:00 and 18:00) highlighting nodes awaiting approval or stuck unschedulable, with ready-to-run query commands. The dual-track "passive query + active push" design balances troubleshooting efficiency and response speed. Kubernetes Events provide a standardized timeline for on-the-spot diagnosis and historical traceability; the alerting system delivers proactive reach from hardware anomaly to remediation result. This "queryable in peacetime, pushed in crisis" approach eliminates traditional ops blind spots and ensures every anomaly in a 10K-GPU steady-state fault environment is instantly sensed, quickly contained, and fully audited. Future work will surface node auto-healing progress in the node pool console for white-screen visibility. Reminder: human fallback only works if critical events actually reach people. Enabling the relevant ACK alert rule sets and verifying notification channels are prerequisites for a closed loop. Observability and alerting run the auto-healing chain in sunlight—issues seen and responded to immediately is the prerequisite for minimizing effective compute erosion. Typical Scenario: Cascaded Card-Level Containment & Node-Level Repair The combined capabilities address high-availability training scenarios via a two-tier cascade: Tier 1: Card-level containment. Device Plugin, based on detection results, immediately removes the faulty GPU from allocatable resources (if user explicitly enabled card-level isolation), stopping new tasks from landing on the bad GPU—fast, localized, shrinking blast radius to a single card. Tier 2: Node-level repair. Detection signals convert to node state; the decision layer generates a repair plan; the node undergoes the full quarantine-drain-repair-validate-un-quarantine loop, truly resolving the hardware issue. Each tier plays its role: card-level "speed" for instant containment and blast-radius control; node-level "thoroughness" for complete repair and validation before return. Both are tunable—users can enable card isolation only for the most severe fault classes, and configure different node-level repair intensities per node pool. This tiered handling minimizes fault impact on effective compute in both time and space dimensions. Productization: Out-of-the-Box Multi-Platform Auto-Healing Even the most complete capability fails if adoption cost is high. ACK packages the system as a console-ready product with three highlights for platform teams: Orchestratable remediation strategies: users define per-fault repair flows—combining quarantine, drain, repair, un-quarantine as needed, with per-step approval gates. Policies are abstracted as reusable auto-repair rules, decoupled from node pools; one definition, many uses; different pools bind different policy intensities. Zero-modification onboarding, consistent experience across shapes: after rule definition, enable node auto-healing in the node pool's managed configuration, attach a rule—effective immediately for both existing and new pools, no business code changes. Lingjun managed pools, ECS managed pools share the same experience; ECS pools can additionally enable active probing to trigger ECS system events that then drive the configured auto-healing flow. End-to-end controllable, observable, fallback-ready: flow state pushed continuously via message center and alerting center; approval-gated steps wait for authorization; failures alert immediately and hand off to humans. Productization turns the ability to guard effective compute from a craft dependent on few experts' experience and on-call into a trustworthy, controllable switch anyone can flip. Building a complete fault auto-healing system—covering heterogeneous detection, event conversion, flow orchestration, tiered authorization, robust execution, and full-chain observability—is a long-term systems engineering investment. Offering it as a platform capability means every ACK team stands on this accumulation, focusing energy on models and business, not re-building foundational stability infrastructure. Landing Practice Recommendations Base approval strategy on fault reversibility: if the business can tolerate drain/interruption, go fully automatic. The more critical the business and the more pre-interruption logic it needs, the more approval gates—recommend approval before quarantine or repair to give a checkpoint window. Ensure alert channels are live: human fallback lives or dies by whether key events reach humans. Enable alert rule sets and verify channels before go-live; otherwise approval reminders and repair failure alerts go unseen, voiding controllability and fallback. Leverage training platform integration: if a custom training platform exists, invest modest integration effort to implement "sense auto-healing signal → auto checkpoint → approve." This converts approval wait overhead into a progress-saving window—the highest direct effective compute gain. Observe first, converge later: start with a conservative "quarantine (approval) + drain + repair + un-quarantine (approval)" policy, observe fault distribution, auto-healing behavior, and business integration for a period, then gradually open mature low-risk paths to full or semi-auto. Trust builds value incrementally. Mine auto-healing data: structured repair records are a goldmine. Regularly review top fault frequencies, node models/batches with elevated failure rates, longest repair durations—feed back into hardware selection, capacity planning, and vendor management. Turn every fault into a data point, moving stability governance from reactive response to proactive optimization. Comparison with Manual Operations Returning to the opening scenario—a GPU emits a severe error. Side-by-side comparison across every phase shows the value concentration: not just saved per-incident toil, but scale and consistency . When nodes fail daily, auto-healing compresses every handling to the shortest, most standard path, eliminating human variability from fatigue and experience gaps. Consistently doing it right and fast is the scarcest capability in large clusters—and it ultimately materializes as higher effective compute throughput. Deeper down, the comparison reveals an operational paradigm shift: from "human-centric, tools-assisted" to "automation-centric, human-as-fallback." Engineers move from per-node firefighters to auto-healing policy designers and exceptional-case arbiters. Human value elevates to defining "what automates, what requires human, what counts as recovered," offloading tedious, repetitive, error-prone execution to the system. This is both an efficiency gain and an engineer experience improvement. Boundaries: What Auto-Healing Explicitly Does Not Do It does not pursue "zero human." For irreversible high-risk actions, it defaults decision rights back to humans; for unconfirmable recoveries, it times out to manual, never gambling on re-admission. Automation's goal is to free humans from repetitive, standardizable handling—not to exclude them from critical decisions. It does not replace job-layer fault tolerance. Auto-healing solves "bad nodes at the infrastructure layer"; training progress protection and job elasticity still require checkpoints, elastic training, etc. Auto-healing provides a clear, programmable intervention moment for them, but does not usurp their role. It does not promise all faults auto-heal. Hardware repair has inherent latency; some complex faults still need humans. Auto-healing's value lies in doing "automate whatever can be automated, notify and make controllable whatever needs humans" to the extreme—not manufacturing a "fully automatic" illusion. Evolution: Toward a User-Extensible Auto-Healing Foundation ACK's auto-healing system currently ships with a suite of built-in heterogeneous detection items and remediation strategies covering the full chain from card-level isolation to node-level repair. Detection precision, conversion rigor, orchestration flexibility, authorization restraint, execution robustness—these five pillars form a stability foundation that is both automatic and controllable. The next direction is openness : allow users to define custom fault types based on their own node states, and reuse the "orchestratable repair flow + tiered authorization" engine to compose the most business-fitting auto-healing policies. Auto-healing will no longer be limited to platform-built-in detection items, but become an extensible, growable stability foundation. This evolution matters because every business defines "what is a fault" differently: some care about specific framework process health, some about custom metric thresholds, some about business-semantic "stuck." The platform cannot enumerate all scenarios, but can distill the hardest, most universal parts—heterogeneous detection, event conversion, flow orchestration, tiered authorization, robust execution, full-chain observability—into a reusable engine, and hand the last mile of "defining what constitutes a fault" to those who know the business best. Platform does deep common capabilities; users do precise business judgment; together they make auto-healing truly fit wildly diverse real needs. In an era where large-scale heterogeneous compute is the core element of AI competitiveness, stability is no longer a background metric but a key variable determining compute ROI. What ACK's fault auto-healing system does is precisely to systematize, automate, and productize this stability, presenting it to users and the ecosystem through Kubernetes as the unified interface—converging heterogeneous underlying complexity beneath the interface, exposing stability capabilities in the most cloud-native way to be used, integrated, and extended. Making every unit of compute spend on effective output as much as possible—that is this system's ultimate meaning. Conclusion ACK's fault auto-healing system for heterogeneous training answers a concrete engineering question: when diverse substrates like Lingjun, ECS, and EGS—each with distinct fault semantics—are stacked in the same cluster, how to distill a node stability governance chain that has automated closed-loop capability while retaining controllable intervention points. The chain's starting point is converging heterogeneous fault signals into Kubernetes Node Conditions. Seemingly just a semantic translation, this is a watershed design decision—it decouples detection from execution, using native API objects as the sole contract. On this unified semantics, the auto-healing flow is explicitly split into quarantine, drain, repair, un-quarantine phases, each extensible with custom actions; critical steps use node labels as authorization credentials, proceeding only after confirmation. This "customizable auto-repair flow + critical-phase human approval" approach aligns with domestic enterprises' controllability requirements. Automation also needs guardrails. The auto-healing engine sets two safety valves at the node pool scope: a max parallel repair cap constraining instantaneous remediation scale, and a faulty-node-ratio threshold triggering circuit break when the pool-wide anomaly proportion exceeds limits. Together they contain the runaway risk of auto-healing under abnormal inputs—rate limiting caps repair throughput; circuit breaking provides an emergency brake; jointly they confine chain reactions from common-mode infrastructure faults, monitoring false positives, etc., within a controlled envelope. Matching this is full-chain observability built on Kubernetes Events as a unified event bus. Every state judgment, approval request, and remediation action is recorded as an event on the corresponding node object—inspectable via kubectl on the spot, and subscribable/consumable/re-orchestrated by upper platforms. Observability here isn't an add-on; it's the evidence base that makes all the above mechanisms trustworthy and auditable. Reference [1] Custom Auto-Repair Flow: https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/custom-node-auto-repair