Design, Implementation, and Optimization of an Android UI Automation Solution

Based on the testing‑pyramid concept from *Succeeding with Agile*, the author outlines a balanced testing strategy (70% unit, 20% API, 10% UI) and highlights the high cost and instability traditionally associated with UI automation.

For the "Youdao Premium Courses" Android app, frequent weekly releases require rapid regression of core cases, making a lightweight, maintainable UI automation approach essential.

The research phase categorizes UI automation work into five areas—case construction, verification, modification, playback, and result handling—and evaluates existing tools such as Selenium, Smart Auto, and Espresso, identifying their limitations for the team’s needs.

The final solution adopts four key components:

**Case Construction**: Extend the open‑source tool Solopi to enable pure‑phone recording of gestures without writing code.

**Case Verification**: Capture screenshots at critical steps and compare them with reference images using OpenCV, eliminating the need for fragile rule‑based checks.

**Result Playback**: Build a test‑report platform that persists playback results, supports annotations, and allows manual review.

**Data Management**: Record associated API calls and cleanup steps, package them with the UI actions into a JSON‑based case, and store everything in a central UI information management page.

During execution, an EnvController prepares the environment, an OrderController drives the sequence of API calls and Solopi playback, and a ReplayCore collects screenshots for OpenCV comparison. Results are visualized with colored markers and can be re‑run or manually annotated.

To guarantee ordered execution, the system writes the current step index to a NowOrder file on the device; Solopi notifies the controller upon completion, enabling synchronous progression and easy recovery from interruptions.

Image‑comparison initially used MD5 hashes, but because UI differences are often minor, the team switched to OpenCV, cropping system UI bars and drawing red bounding boxes around mismatches. An algorithmic improvement later prevented overlapping boxes from obscuring the screenshot.

The deployed solution has been in production since April 2021, covering 95% of core cases for the premium‑course app and 70% for the translation app, reducing manual regression time from 4 hours per release to 0.5 hours per release.

Future work includes adding Windows support, exposing failed steps directly in the platform, allowing configurable ignore‑areas for image comparison, and extending the framework to additional business lines.