Design and Implementation of the AutoMotion UI Automation Testing Platform

UI automated testing is essential for product quality, especially under the pressure of cost reduction and efficiency improvement. However, writing and maintaining test scripts is labor‑intensive, and frequent UI changes and unstable network data often cause flaky test results.

To address these challenges, the team built the AutoMotion platform, which lowers the entry barrier for UI automation, improves usability, and adds two key capabilities: test‑case self‑healing powered by large language models (LLM) and a data sandbox that isolates and controls test data to ensure consistency and repeatability.

Background

The team previously tried common tools such as Cypress and encountered four major pain points:

High cost of hand‑written scripts.

Unstable test results caused by changing network data.

High maintenance cost due to frequent project iterations.

Difficulty integrating automated tests into standard CI/CD pipelines.

Core Mechanisms

1. Test‑case recording and generation : A Chrome extension records user actions (clicks, inputs, scrolls, drags) and automatically generates Cypress scripts. The extension also captures storage, cookies, window size, and network requests, converting everything into a structured DSL.

2. Data sandbox : During recording, the extension collects all external data (API responses, localStorage, cookies, viewport size). When a test runs, the sandbox replays these inputs 1:1, isolating the test from external variability. Users can selectively disable parts of the sandbox (e.g., keep cookies for login).

3. Visual editing and self‑healing : Small updates can be made via a visual editor; larger changes can be edited directly as Cypress code. When a selector fails, the platform pauses execution, sends the DOM snapshot and the outdated selector to an LLM, which returns an updated selector. The test case is then patched and re‑executed.

4. Open API : Each recorded test case is exposed as an OpenAPI endpoint, allowing seamless integration with GitLab pipelines, CI/CD tools, CLI utilities, or scheduled jobs.

Technical Highlights

Recording pipeline : The Chrome extension injects a JavaScript snippet into the page header to capture user actions, storage, and cookies. It also uses the DevTools protocol to intercept network traffic. After recording, the collected data is transformed into a DSL JSON format.

Build & update process : Webpack builds the DevTools UI, other files are processed for environment variable substitution, the manifest version is bumped, the project is zipped, uploaded, and the version is stored in a database. Users are prompted to update when a new version is available.

Execution engine : A Node.js service spins up a Cypress environment inside a container. The DSL is translated into executable Cypress scripts, which run concurrently across multiple instances. Results are reported back to the front‑end.

Event listening code :

window.addEventListener('input', this.handleInput, true)

Enhanced addEventListener wrapper :

const originalAddEventListener = window.Element.prototype.addEventListener;
// Replace all elements' addEventListener
window.Element.prototype.addEventListener = function(type, originalListener, ...others) {
  const listener = function(event) {
    if (isRecording && ['mouseenter','mouseleave','mouseover','mouseout','mousemove'].includes(event.type) && event.target === this) {
      handleMouseAction(event);
    }
    originalListener.call(this, event);
  };
  return originalAddEventListener.call(this, type, listener, ...others);
};

Selector example before and after UI change :

cy.get('.task > .btn').click()

When the UI adds an extra wrapper ( .task-content ), the original selector fails. The platform can either use a more robust selector like .task .btn or rely on LLM‑driven self‑healing to generate .task > .task-content > .btn .

Target element identification

The platform first tries basic CSS paths (e.g., body > div > #app > div.app-right > div:nth-child(2) > div.text ). To improve robustness, it extends selectors with class and index information (e.g., xx > div.btn.primary:nth(2) ) and optional text matching ( :contains('Export') ). When these still fail, an LLM processes the compressed DOM (after removing scripts, styles, whitespace, and keeping only tag, id, class, text) and suggests a corrected selector.

Data sandbox – network request handling

During recording, all API calls are captured via chrome.devtools . At execution, requests are matched by URL, method, body, and temporal proximity to the triggering UI action. A similarity‑based matching algorithm selects the best‑fit recorded response; unmatched requests fall back to real backend with recorded cookies to avoid authentication failures.

Multi‑environment and concurrency

Runners (containers with headless browsers and Cypress) are deployed per environment (dev, test, prod). The server distributes test cases to the appropriate runner, which queues them and respects a maximum concurrency limit.

DSL and script export

The DSL is a JSON representation of a test case, enabling visual editing, storage, and conversion to Cypress scripts. Users can export the script, modify it locally, and re‑import it as a “script‑type” test case.

Future outlook

Current workflow still requires manual recording. The team envisions fully automated test generation by feeding product requirements to an LLM, or by mining real user interaction logs to automatically derive high‑frequency user flows as test cases.