Boosting Web UI Test Efficiency with AIGC: From Manual Scripts to Intelligent Automation

Business Background

Tencent Advertising Delivery System is a core platform handling billions of ad revenue, requiring high quality assurance. The Web UI testing process needs optimization to improve efficiency and quality.

Web UI testing current status

Resource consumption: UI testing accounts for 60% of total testing effort, with regression 10% (industry 20‑35%).

Defect detection: contributes over 65% of total defects.

Historical technical limitations: traditional UI automation ROI insufficient, leading to a long‑standing gap.

Web UI testing efficiency pain points

High maintenance cost of test cases due to frequent page changes.

Unstable element locating methods (XPath, coordinates, dynamic IDs).

Additionally, UI automation requires stable environments, has low execution efficiency, and difficult debugging, further reducing confidence.

Solution Overview

1. Traditional Solutions

Industry mainly uses script‑based tools like Selenium and Playwright. Record‑playback tools are less flexible and fragile. Some attempts at keyword‑driven or data‑driven testing exist, but script development remains dominant.

Key engineering solutions:

Improve script writing efficiency

Page Object Model (POM) and modular design to separate page operations, component logic, and business flow, enhancing code reuse.

Data‑driven testing to separate test data from scripts, reducing maintenance.

Address element locating stability

Composite locating strategy using CSS selectors with attribute filtering and relative positioning.

Element fingerprinting by hashing element features to generate stable identifiers.

Despite these, challenges remain: manual script maintenance, sensitivity to UI changes, and limited automation of large‑scale scenarios.

2. AIGC Innovative Solutions

Large language models enable semantic test scripts that are more stable than code. By maintaining semantic scenarios, scripts can be auto‑generated, improving scalability.

Examples of AIGC‑driven UI testing tools include LLM automation for app/desktop UI, UAT for WeChat Pay, AppAgent, MobileAgent, AUITestAgent, Browser Use, Playwright MCP, etc.

3. Initial Experience with AIGC

Experiments with Browser Use and Cursor+Playwright MCP show AI can replace manual or traditional scripts to some extent, but challenges remain:

Accuracy below 50% without domain‑specific training.

Data security and compliance when sending ad data to external services.

Low efficiency of assertion generation.

Thus a systematic solution is proposed.

Hybrid Model‑Driven Automation Framework

The chosen solution uses Tencent's internal Hunyuan large model as the core engine, forming a two‑stage pipeline: test case generation and execution.

In the generation stage, a textual scenario is transformed into an executable script with assertions; the execution stage runs the script and produces a report.

Each test case consists of tasks, each task contains steps. Agents:

Decision Agent : decides the next operation based on page state, task requirements, and history.

Reflection Agent : checks whether the performed operation matches expectations.

Verification Agent : generates or executes assertions.

Perception Layer: Web Understanding

Accurate page understanding is essential. Selenium extracts page information, and visual annotations highlight interactive elements. Enhancements include adding textual information, hierarchy, and element state to improve model comprehension.

Challenges such as overlapping elements, occlusions, and small fonts are addressed by supplementing text and hierarchical data.

Decision Layer: Task Planning and Operation

Decision and reflection agents form the system's brain, using Hunyuan‑7B‑MoE (130 B active parameters) fine‑tuned with SFT. The process includes observation (reducing hallucinations) and thinking (task decomposition, memory of past actions).

Verification Layer: Intelligent Assertion Generation

Assertions are generated based on UI interaction, front‑end output parameters, and database fields, using a DIFF‑based approach to compare expected and actual states.

Data‑Driven Model Iteration

Training data is built automatically, covering component and page coverage, CoT enhancements, and semi‑automatic data generation using Selenium IDE recordings and teacher models.

Model evaluation shows accuracy improvements from <50% to over 90% after iterative training.

Flow Recording‑Based Test Evolution

User interaction logs are recorded, parsed into structured scripts, and used to auto‑update Selenium scripts when UI changes occur.

CI/CD Integration

The automation pipeline consists of test case generation (manual or scheduled) and test execution integrated into the CI/CD workflow, providing quality gates for code merges.

Results and Future Plans

After three months, over 500 core business test cases were generated, achieving >90% script generation accuracy and enabling dynamic test case evolution via flow recording. Future work includes enhancing web understanding, complex component handling, RAG‑enhanced scripts, fully automated model training, and online user operation recording.

Acknowledgments

Thanks to collaborators from Shanghai Jiao Tong University and all team members for their contributions.

{
  "pages": "...",
  "actions": [
    {
      "action": "click",
      "element": {
        "role": "button",
        "name": "品牌宣传",
        "state": {"disabled": false, "selected": false},
        "hierarchy": [
          "button[@class='l-scenes-btn flex-center flex-col']",
          "div[@class='grid gap-8 l-scenes']",
          "div[@class='meta-content']"
        ]
      }
    }
  ]
}

const handleEvent = (event: Event) => {
  // 获取事件目标元素
  const targetElement = event.target as HTMLElement;
  // 初始化一个数组来存储事件路径中的元素
  const elements: HTMLElement[] = [];
  // 当前元素从事件目标开始
  let currentElement: HTMLElement | null = targetElement;
  // 遍历事件冒泡路径，直到 document
  while (currentElement) {
    elements.push(currentElement);
    currentElement = currentElement.parentElement;
  }
};