Boost Frontend Testing with Midscene.js: AI‑Powered UI Automation Made Easy

On March 8, at the 19th D2 Terminal Technology Conference, Ma Xiaofeng presented "midscene.js: AI in Front‑End Testing".

Midscene.js is an open‑source UI automation tool that incorporates multimodal AI reasoning to simplify the creation and maintenance of UI tests.

Start with the Chrome Extension

Before building a full test project, you can try the Midscene.js Chrome extension, which requires no code.

For example, to test Midscene on an e‑commerce site, follow these steps:

Install the Midscene extension and configure the AI service key.

Open the target e‑commerce site.

Enter interaction commands in the extension, click “Run”, and view the execution result and playback animation.

Click “Report File” to obtain a complete replay file that shows all AI steps and reasoning, useful for later reference.

In the Query panel, describe the desired UI content and format, click Run, and receive the corresponding JSON response.

The Assert panel provides assertion capabilities.

Three Key Interfaces

The extension exposes three essential AI‑driven methods: .ai .aiAction – Describe steps and execute interactions. .aiQuery – Extract data from the UI, returning JSON in any desired structure. .aiAssert – Perform assertions.

These interfaces become intuitive with AI assistance.

Integrate into Code: JavaScript or YAML

After trying the extension, you can integrate Midscene.js into your own test projects using several approaches.

YAML Scripts

YAML scripts are readable and require no large test framework, suitable for simple inspections and validations.

Example script:

After setting environment variables, run the script with a single command.

JavaScript SDK for Playwright or Puppeteer

Midscene provides a JS SDK that can be integrated into existing Playwright or Puppeteer scripts.

Example integration:

Bridge Mode

Bridge mode lets local scripts control the desktop Chrome version, reusing cookies, login state, and UI context to start automation without setting up an environment.

Midscene.js Project Information

GitHub repository: https://github.com/web-infra-dev/midscene

Example project: https://github.com/web-infra-dev/midscene-example

Homepage and documentation: https://midscenejs.com/zh

D2 Sharing

Midscene.js is an open‑source UI automation tool from ByteDance’s Web Infra team. At the 19th D2 conference, AI lead Ma Xiaofeng presented AI‑driven frontend testing, covering industry trends, UI automation architecture, model selection, and practical deployment paths, inviting discussion on testing efficiency in the AI era.

We look forward to seeing you in Hangzhou on March 8.