How Large‑Model AI Can Revolutionize UI Automation Testing

Background

Traditional UI automation relies on DOM element locators and hand‑written scripts. Frequent UI changes, the need for separate scripts per platform (Web, H5, iOS, Android), and difficulty recognizing dynamic or visual elements lead to high maintenance cost and low adaptability, which hampers continuous delivery and left‑shift testing.

AI‑augmented UI automation approach

Intelligent visual element recognition : a vision‑language model (Qwen2.5‑VL) identifies text, icons, images and remains robust to UI changes.

Cross‑platform universality : screenshot‑based input enables a single script to run on any UI that can be captured.

High precision and robustness : the model handles dynamic, blurred or complex visual content.

Readable, maintainable scripts : natural‑language descriptions replace low‑level locators.

Common UI issue detection : custom prompts surface style problems such as white‑screen, overlapping elements, NaN/Null values.

Model‑driven planning : a ReAct‑style architecture iteratively decomposes user intents into actions.

Solution selection

Two stacks were compared in February: OmniParser + Qwen2.5‑VL + Playwright and Midscene + Qwen2.5‑VL + Playwright . Both demonstrated strong visual‑language capabilities, but the final decision favored the Qwen2.5‑VL + natural‑language description → Playwright combination because it maximizes the model’s core ability, reduces integration complexity, and provides a clean “brain‑limb” separation.

Key design decisions

Direct model input : feed the browser screenshot and a user instruction (e.g., “click Submit”) directly to Qwen2.5‑VL, letting the model output the target element and coordinates without a separate annotation module.

Decoupled “brain” and “limb” : Qwen2.5‑VL generates intent and coordinates; Playwright executes the actions, allowing independent scaling and maintenance.

Visual‑driven cross‑platform support : screenshot‑based processing works for any UI that can be captured, fulfilling the “write once, run everywhere” goal.

System architecture

The platform consists of three layers: an AI Agent layer, a scheduling layer, and a Playwright execution engine. It supports both PC and APP automation.

Real‑time interaction & single‑step debugging

Remote Playwright execution is visualized in a local browser, allowing bidirectional mouse/keyboard control with millisecond latency. Each step records a screenshot and status, simplifying failure analysis.

Scheduler

A micro‑service orchestration engine provides dynamic load‑balancing, distributed locks, priority‑based auto‑scaling and fault‑tolerance, enabling tens of thousands of concurrent test cases.

AI Agent

The Agent abstracts model invocation, prompt management, retrieval‑augmented generation and history storage. It receives a pre‑execution screenshot and a textual instruction, then returns a structured action JSON for Playwright.

{"action":"click","coordinates":[x,y],"text":"Submit"}

Additional fields such as reasoning can be included for debugging.

Playwright execution framework

Each test case receives an isolated Playwright instance (browser + renderer processes) to guarantee atomicity. Supported actions include navigation, click, hover, fill, drag‑and‑drop, file upload, screenshot, etc. Instance limits, mode‑specific caps (debug vs. non‑debug) and custom load‑balancing ensure resource stability.

Technical challenges

Visual recognition accuracy & robustness : dynamic, highly similar or heavily customized UI elements may cause mis‑identification or false positives.

Mapping model output to Playwright actions : converting bounding‑box coordinates to precise click points and selecting the correct action type (click, fill, double‑click, etc.).

User intent understanding : ambiguous or colloquial instructions require disambiguation and multi‑step planning.

Performance and latency : large‑model inference introduces delay; GPU utilization and batching must be optimized.

Debugging the AI black box : failures can stem from model errors, prompt issues, or Playwright execution, demanding comprehensive logging and visual trace tools.

Mitigation strategies

Agile iteration, continuous prompt refinement, strict JSON output contracts, fallback heuristics, and a visual debugging console were introduced to improve stability, reduce hallucinations and accelerate failure diagnosis.

Results and outlook

After four months of MVP rollout, the platform executed >4,400 UI test cases (including >3,000 in an automated lab), discovered 248 defects (content, style and data issues), and achieved a daily pass rate of 95.5 %.