AI UITester: The New AI‑Native Paradigm for Visual UI Automation Testing

Why AI‑Native UI Testing Is Needed

Traditional UI automation faces three major pain points: high migration cost from descriptive test cases to executable scripts, low debugging efficiency with heavy manual intervention, and duplicated maintenance for iOS, Android and HarmonyOS scripts.

Capability 1 – Automatic Conversion of Test‑Case Platform Data

ai_uitester builds an automated pipeline that transforms the JSON exported from a test‑case platform into executable steps.

Platform JSON Export ↓ Phase 1: flatten tree – extract leaf nodes and breadcrumb paths ↓ Phase 2: parse cases – breadcrumb → structured case data ↓ Phase 3: deduplicate – remove existing suite cases ↓ Phase 4: LLM enhancement – generate executable steps + inject Wiki knowledge ↓ Phase 5: persist – write configuration files ↓ Phase 6: archive – record version history

The LLM enhancement module converts descriptive checkpoints (e.g., “a list displays correctly and can be scrolled”) into a JSON script containing tap, wait, assertion, swipe actions.

Prompt Engineering for High‑Quality Steps

StepGenerator uses carefully crafted prompts with four constraints:

Each step performs a single action; the screen state is refreshed after every action.

Strict element‑matching rules: MatchedText must be copied character‑by‑character from screenshots; if confidence ≤ 0.5 the action is null.

High‑priority knowledge (pop‑ups, permissions, login) is auto‑injected by the VLM.

Platform‑specific adaptation: prompts switch system commands according to the target OS.

Parallel Enhancement and Checkpoint Resume

LLM enhancement runs in parallel, pre‑loading Wiki knowledge per module to avoid redundant calls. After each pipeline stage a checkpoint file is written; on interruption the pipeline resumes from the last completed stage. If enhancement fails completely, a fallback case is generated using a multi‑level degradation strategy.

Wiki Knowledge Base

The Wiki is embedded across core modules and consumed in five scenarios: case enhancement, self‑healing diagnosis, runtime execution, downstream skill consumption, and feedback loop logging. Its quality directly influences three core metrics (efficiency, coverage, reliability).

Capability 2 – AI Intelligent Debugging and Self‑Healing

Traditional debugging loops through screenshots, manual cause analysis, script modification and re‑execution, often requiring many iterations.

ai_uitester adds an AI debugging mode:

Case execution (while loop, supports dynamic step changes) ↓ Step failure ↓ Failure classifier (rule engine) → device/timeout/network → auto‑retry; business → AI diagnosis ↓ AI diagnosis (VLM) receives labeled steps, error info, failure screenshot, Wiki knowledge → outputs diagnosis, confidence, repaired steps, resume index ↓ If confidence ≥ threshold → auto‑apply fix; else → human review.

The failure classifier filters out non‑business failures (device, timeout, network) before invoking AI diagnosis.

Five root‑cause categories are identified, and three real self‑healing cases are presented:

UI change – button moved (confidence 0.9): original tap fails, repaired by tapping the new location.

UI change – extra action required (confidence 0.8): insert a wait before re‑tapping the entry point.

Pre‑step failure – modal blocks subsequent steps (confidence 0.9): insert conditional action to dismiss the modal.

Confidence Mechanism

The system prefers omission over mis‑click: “matched text must be copied exactly; if confidence ≤ 0.5 the action is null.” This reduces the risk of erroneous clicks, which are far more damaging than missed clicks.

Capability 3 – VLM‑Driven Cross‑Platform Unification

The core execution model is a closed loop: Screenshot → Understanding (VLM) → Execution . Because the VLM works on pixel‑level screenshots, the same JSON script runs unchanged on Android, iOS and HarmonyOS.

The unified API covers actions, assertions, queries and waits, abstracting away platform differences.

BaseAIDriver implements the loop (up to 20 rounds) with confidence‑calibrated clicks, knowledge‑base queries and forced continuation after knowledge lookup.

Architecture Design Trade‑offs

Step‑wise execution is chosen over a single‑plan approach because UI state is nondeterministic; each step must be based on the latest screenshot. The confidence threshold of 0.5 balances accuracy and coverage, as determined by extensive empirical tuning.

Self‑healing returns a complete step list rather than a diff to avoid index drift after multiple repairs, accepting higher token cost for reliability.

Industry Comparison – Why This Is a New Paradigm

Three routes are compared across core technology stack, cross‑platform ability, maintenance cost and self‑healing:

Traditional (Appium/Selenium/XCUITest): element‑locator based, fast, mature, but requires three separate scripts, breaks on UI changes, linear maintenance cost, no self‑healing.

AI‑assisted (Test.ai / Applitools): adds AI element‑matching or visual diff, reduces locator maintenance, still relies on element detection, limited cross‑platform adaptation, only element‑level self‑healing.

AI‑Native (ai_uitester): replaces element locators with VLM‑driven visual understanding, provides full‑process self‑healing, unified JSON across platforms, and a knowledge‑base that decouples business rules from execution.

Business Results

Deployed in the 得物 App client, ai_uitester achieved the following average improvements:

Efficiency metrics (e.g., test case conversion time, debugging cycles) dropped by up to 70 %.

Quality metrics (defect detection rate, regression coverage) improved by 30‑40 %.

Self‑healing success depends on Wiki quality; complex workflow changes still require human confirmation.

Conclusion

ai_uitester exemplifies a shift from “code‑driven, manual‑debug” testing to a “visual‑driven, AI‑self‑healing” paradigm, unifying three mobile platforms under a single abstract model and continuously improving through a closed‑loop knowledge base.