AI-Powered Visual Defect Detection for Mobile App UI Testing: Methodology, Data Construction, Model Training, and Evaluation

The article introduces an AI‑based visual testing approach to detect UI defects such as blank screens, overlapping text, and image missing issues in large‑scale mobile apps, explaining the background, business challenges, and the shift from traditional code‑intrusive testing to non‑intrusive, image‑driven methods.

It outlines three practical detection strategies: (1) using GUI tree node information, (2) image feature matching with algorithms like SIFT, SURF, ORB, and (3) deep learning model training for end‑to‑end classification and bounding‑box detection.

Data collection involves building a comprehensive image dataset from the app’s pages, generating negative samples for defects, and labeling each image with class and defect position coordinates for supervised learning.

The chosen model is a ResNet‑18 network enhanced with a Squeeze‑Excitation (SE) layer; detailed architecture diagrams and parameter counts are provided, showing a total model size of 58.54 MB.

Training follows standard deep learning practices: dataset split, hyper‑parameter setup (epochs, learning rate, batch size), loss function (Cross‑Entropy), and iterative optimization. After several epochs, the best model is selected based on validation performance.

Evaluation results include ROC curves, confusion matrices, and heat‑map visualizations of model attention, demonstrating high detection accuracy on both test and real‑world samples, while also analyzing failure cases such as mislabeled data and missing defect types.

Future work focuses on expanding the negative sample pool, exploring unsupervised anomaly detection to reduce labeling effort, and further optimizing the model and engineering pipeline to enhance AI‑driven testing capabilities.

Figure 7: Randomly sampled annotated examples.

Figure 12: Model detection heat‑map highlighting defect regions.