A One‑Stop UI Automation Testing Framework for NetEase Cloud Music

Background: NetEase Cloud Music is a large music platform that also provides live streaming, K‑song, mlog, and long‑audio services. With more than 3,000 P0/P1 test cases and a bi‑weekly iteration cycle, each iteration only leaves about 1.5 days for regression testing. To cope with limited time, the team selects a subset of core scenarios for regression, which sometimes leads to low‑level bugs reaching production.

The testing team evaluated existing UI automation frameworks but found them lacking in case‑entry cost, stability, execution efficiency, and success rate. Consequently, they decided to design a custom automation framework tailored to Cloud Music’s needs.

Core Concerns

Case entry cost : The volume of cases is huge and will keep growing, so low‑cost entry is essential. Existing approaches include high‑level scripting languages, natural‑language descriptions, IDE tools, and record‑and‑play (operation‑as‑case). The record‑and‑play method offers the lowest entry cost.

Execution stability : Cases should remain stable across version changes. Android can use element IDs, while iOS often relies on XPath, which is fragile.

Execution efficiency : Time is spent on command transmission (e.g., WebDriver‑based C/S models) and element locating (black‑box page dumps).

Coverage & success rate : The framework must cover a high proportion of cases and achieve a high pass rate, requiring auxiliary capabilities such as network mock and multi‑account support.

Industry Framework Comparison

A table compares UIAutomator, XCUITest, Appium, SmartAuto, AirTest, and Solopi across entry cost, stability, efficiency, and platform support. No single framework satisfies all Cloud Music requirements, prompting a self‑built solution.

Solution Overview

Case entry cost : Adopt Solopi’s record‑and‑play approach for Android and develop a similar capability for iOS.

Execution stability : Leverage the internally developed “Dawn” telemetry points that bind UI elements on both Android and iOS. When Dawn points exist, they are used for precise locating; otherwise, Android falls back to element IDs and iOS to XPath.

Execution efficiency : Use Dawn points for white‑box element location and transmit whole test cases as a single payload to avoid per‑command network overhead.

Coverage & success rate : Provide surrounding capabilities such as network mock (contract platform), account level simulation, and pre/post‑condition binding to prepare the execution environment automatically.

Design Details

Dual‑Platform Case Reuse

Define a JSON‑based test case format that can be edited and is readable, enabling reuse across Android and iOS.

Android Design

Reuse Solopi’s recording and playback engine, integrate Dawn point logic, and add white‑box SDK communication with a separate test‑runner app. Architecture diagram is shown in the source.

iOS Design

Build a custom framework from scratch because no suitable record‑and‑play solution exists. Hook system APIs for tap, double‑tap, long‑press, swipe, and capture keyboard input via external tools. Playback uses private UIEvent APIs. An SDK is embedded in the test app, and a local service handles communication with the platform.

Platform (Saturn)

Saturn provides a unified UI for creating, editing, and executing test cases, managing execution machines, custom devices, and scheduled tasks.

Mock Capability

Integrate with a contract‑based mock platform that allows QR‑code‑driven interface mocking at the parameter level.

Private Cloud Deployment

Reference open‑source projects such as STF and open‑atx‑server to build a private device farm that supports both Android and iOS, enabling decentralized, on‑premise execution.

Results

Recording efficiency improved dramatically (see source chart). Execution efficiency also increased, achieving 73% overall case coverage and over 90% success rate for dual‑platform cases. Regression time dropped from 1.5 days with 15 person‑days effort to 0.5 days with 6 person‑days, a >60% efficiency gain.

The framework, named Athena , is now used across all Cloud Music business lines and continues to evolve.

Conclusion

The one‑stop UI automation solution addresses high entry cost, instability, low efficiency, and limited coverage by combining record‑and‑play, Dawn telemetry, white‑box element location, and comprehensive mock capabilities, all deployed on a private cloud device farm.