How AI Self‑Healing Transforms Mobile UI Automation Testing

Background and Challenges

Mobile testing teams need to reduce manual effort. Existing solutions include a zero‑code UI recording‑playback system for legacy regression and a multi‑device synchronous mode for new‑feature testing. With more than 18,000 scripts, frequent UI changes cause script breakage and new‑feature execution remains manual.

Limitations of Traditional UI Automation

Maintenance loop : fragile locators (ID, image, text) break with weekly UI tweaks.

Stability bottleneck : pop‑ups (system prompts, permission dialogs, ads) keep pass rates around 80% and require extensive handling logic.

AI‑Driven Self‑Healing Architecture

AI self‑healing leverages multimodal feature fusion, visual models and semantic analysis, operating in four stages: perception, diagnosis, decision, and repair.

Architecture Overview

Capability layer : smart popup handling, text self‑healing, element self‑healing, intelligent wait.

Data collection layer : captures screenshots, OCR‑extracted text, UI hierarchy; stores a case library mapping scripts to self‑heal actions.

UI self‑healing agent layer : knowledge base, intelligent engine for popup analysis and element perception, decision body that generates repair commands.

Five‑Step Diagnostic Process

Check device health (memory, disk, network, connections).

Verify automation framework availability (session state, driver service).

Use a large multimodal model with popup knowledge to detect and handle interfering pop‑ups.

Apply a visual‑language model to assess page load status and content completeness.

Compare current UI against stored control portraits using multimodal reasoning to pinpoint element changes.

Each step outputs a structured Action command (e.g., “close popup”, “repair locator”) that the engine executes, forming a closed‑loop detect‑analyze‑solve cycle.

Smart Popup Handling Engine

Across Android, iOS and HarmonyOS, pop‑ups vary in layout and wording. The AI engine abstracts visual layout and semantic keywords to build a structured popup knowledge base, enabling robust handling of unseen variants.

Page‑Change Perception Engine

During baseline collection, the system gathers three data streams via adb (Android), tidevice (iOS) and hdc (HarmonyOS): device screenshots, OCR‑extracted text (PaddleOCR), and UI hierarchy (Appium/hdc). Images are compressed and non‑essential regions trimmed to reduce token usage.

A “digital ID” for each control (type, hierarchy, context) is created. The engine uses three‑level matching: semantic similarity, visual bounds alignment, and DOM structural comparison.

Intelligent Repair

Based on confidence scores, the engine emits either selector optimization (adjust existing locator) or selector reconstruction (generate a new robust locator). Output includes change type (text, element, layout), the new locator, and step‑level retry instructions that preserve business state.

Comparison with Traditional Automation

Problem handling : manual fixes vs. AI automatic intervention.

Maintenance cost : manual updates vs. AI‑driven reduction.

Adaptability : fixed scenarios vs. AI can handle unknown changes.

Resource efficiency : full re‑run vs. precise step‑level retry.

Practical Impact

Average script pass rate increased from 80% to over 90%.

325 high‑frequency scripts across driver and user apps successfully self‑healed.

Script‑maintenance effort reduced by approximately 40%.

Future Outlook

Future work includes extending AI self‑healing to full test‑case generation, exposing MCP/OpenAPI services for AI‑driven test agents, and optimizing multimodal pipelines to lower token consumption.