Beyond a Demo: How Huajiao’s QA Team Integrated Real‑Device UI Automation into a Full Regression Workflow

Background

Previously a regression run took roughly two days; after adopting real‑device UI automation the core execution chain can be completed in about ten minutes.

Observability

The execution UI displays steps, timing, replay video, retained parameters, and element locations, eliminating repeated page checks, result reviews, and evidence gathering.

Challenges

Real‑device UI automation is often mistaken for a simple demo—writing a YAML file and connecting a phone. In a true regression workflow, issues arise such as missing environments, incorrect device‑script mapping, multi‑device coordination, ambiguous failure semantics, and the need for result verification.

Engineering Process

We constructed a repeatable, observable, and failure‑aware engineering flow.

1. Pre‑execution Environment Check

Before running the real‑device checklist, we verify base dependencies, device status, and model configuration. The check can run in diagnostic mode or provide installation/fix prompts.

- CheckOnly: diagnose only, no install<br/>- Base dependencies: check Node.js/npm, ADB/platform‑tools, Midscene CLI<br/>- Project dependencies: verify Midscene Android execution deps<br/>- Model config: ensure model‑related env vars, API Key set/not set<br/>- Device status: adb devices -l to confirm device is in <em>device</em> state<br/>- Handling: skip already satisfied components, prompt for missing ones

2. Device Mapping & Multi‑Device Queue

When multiple phones are connected, we list available devices and pending checklists, then let the user explicitly bind each task lane to a specific device, preventing duplicate mapping. The system checks for repeated bindings and ensures each lane runs on its assigned device.

3. Queue Management

The queue can advance by round‑robin, concurrent start, or manual promotion. If one device finishes early, it can wait for others to keep observation consistent, or QA can pause to inspect.

4. Failure, Pause, Rerun, Stop

Failure : record the failure; subsequent actions follow queue rules and QA judgment.

Pause : halt the current execution without treating it as a continuation.

Rerun : restart the script on the current lane after terminating the previous process.

Skip : bypass remaining tasks on the current device while other devices continue.

Stop All : terminate the entire regression when environment or execution conditions become invalid.

5. Result Review

Midscene reports retain execution steps, timeline, replay screenshots, JSON view, parameters, and element location info. QA can trace whether a failure stems from page changes, locator instability, environment issues, or genuine business bugs.

Takeaways

Automation solves repeatable execution and traceability but does not replace QA’s risk assessment. Teams should ensure: environment is checkable, device‑script mapping is confirmable, queue behavior is predictable, process is observable, failures are manageable, results are reviewable, and responsibilities remain clear.