TARS UI Automation System: Metrics, Challenges, and Automated Case Recording Solutions

Introduction

In March 2020, Qunar launched the TARS UI automation system, which by the end of the year achieved its Phase‑1 goals, supporting multiple business lines, releasing over 30 versions per month, executing hundreds of automated runs, and preventing any P1/P2 incidents while saving nearly a thousand person‑days.

Core Metrics and Problems

The team identified five key metrics: coverage, intercept‑skip rate, accuracy, runtime, and bug recall rate. Continuous monitoring revealed low coverage, high skip rates, sub‑80% accuracy, long execution times, and low bug recall, each traced to high case‑writing cost, complex environment setup, and unstable test data.

Automatic Case Recording Solution

UI Automation Techniques

Two element‑location methods are discussed: image‑recognition (flexible but error‑prone) and element identifiers (precise but fragile). The team chose identifier‑based locating for reliability.

Case Writing Approach (POM)

Cases are built on the Page‑Object‑Model, encapsulating element tags and actions to reduce maintenance effort.

Automatic Tag Generation

Tags are generated from component hierarchy (business line|page|module|component|key) during compilation via plugins, ensuring uniqueness and resistance to minor DOM changes.

Automatic Tag Maintenance

When code changes, tags are diff‑matched to associate old and new identifiers, automating updates for unchanged components while flagging deleted or modified ones for manual review.

Recording QA Operations

Interactive element callbacks are wrapped with decorators to log timestamps, element tags, operation type, and code version, producing an operation list that is later translated into automation scripts.

Data Mocking

All request/response data are captured during recording; during playback, mock data replace live calls, guaranteeing stable execution despite backend changes.

Automatic Assertion Generation

Smart OCR identifies UI text within context to create reliable assertions, while key request data nodes are compared against recorded values to validate business logic.

Platform Revamp

The infrastructure was upgraded to a distributed device‑scheduling system built on STF, enabling shared mobile devices across business lines, and a cloud‑based testing platform that supports remote device rental, performance testing, and behavior tracing.

Summary and Outlook

Post‑upgrade metrics show improved coverage, reduced skip rate (<8%), accuracy above 95%, and average task runtime under 10 minutes. Future work includes IDE enhancements and AI assistance to further simplify case authoring and handle edge‑case scenarios.