How AI Can Automate UI Testing: Building Image‑Based Anomaly Detection

Background

As software architecture has shifted from monolithic to AI‑enabled distributed systems, testing practices have evolved from waterfall, server‑side testing to agile, end‑to‑end, full‑link testing that now includes mobile devices. Mobile testing can be code‑intrusive (SDK integration or system hooks) or non‑intrusive (user‑view based, UIAutomator/WDA, GUITree, or image‑based recognition). The emerging "Test By AI" movement aims to boost test path generation, data diagnostics, and assertion accuracy, with platforms such as Test.AI, Applitools, Mabl, AirTest, AppiumPro, Fastbot, SmartX, RXT, DevEco Studio, PerfDog, and GameAISDK.

Pain Points

Large‑scale apps support thousands of services and millions of daily active users across diverse devices, leading to quality‑control challenges such as blank screens, missing images, overlapping text, garbled characters, incorrect currency symbols, and missing fallback graphics.

Feasibility Research

Manual identification of these defects is costly and error‑prone. Three AI‑enabled solutions are explored:

Using GUITree node information to verify element existence and attributes; limited to detectable UI components.

Comparing image features (SIFT, SURF, ORB) to pre‑defined error templates; requires a curated error‑image library and incurs linear computational cost.

Training a deep‑learning model on a large labeled dataset to perform one‑shot classification and threshold‑based judgment, supplemented by secondary checks for higher accuracy.

Anomaly Construction Capability

To generate negative samples, the article demonstrates two concrete techniques:

Missing Image

Identify image elements, binarize them, locate contours, and replace the pixel region with a white rectangle; optional dilation or erosion improves edge definition.

Overlapping Text

Detect a text region via OCR combined with CV, extract its background color and font size, then programmatically overlay the same text to create overlap; this method produced a dataset of tens of thousands of overlapping‑text images with minimal manual effort.

Algorithm Development Practice

Typical workflow includes:

Development environment setup (local GPU machine vs. cloud services such as AWS or JD Cloud; primary languages Python or C++).

Sample data preparation (collecting, cleaning, labeling, augmenting; ~70‑80% of effort). Normal samples are labeled -1, blank‑block as 0, overlapping‑text as 1.

Network design (CNN‑based; ResNet‑18 with SELayer chosen for this project).

Model training and testing (train/validation split, hyper‑parameters like epochs, learning_rate, batch_size; loss function = CrossEntropy after softmax).

Algorithm iteration and optimization (heat‑map analysis, bad‑case review).

ResNet‑18 with SELayer architecture:

Training results show a clear ROC curve separating test and real samples.

Heat‑map of the fourth ResNet layer highlights strong responses on blank‑block defects, confirming learned feature relevance.

Bad‑Case Analysis

Case 1: Model marked a truly abnormal sample as normal; further inspection revealed a labeling error, suggesting the model can help auto‑detect mislabeled data.

Case 2: Model missed a blank‑block defect because the training set lacked “pink‑background” examples; future data collection will target this gap.

Future Planning

Building on successful AI‑driven detection, the roadmap includes deeper data cleaning, model refinement, exploration of unsupervised anomaly detection, strengthening double‑check mechanisms in production, and continuous extraction of bad‑case insights to improve AI judgment accuracy across test path generation, data diagnostics, and assertion validation.