From Selenium & Appium to Skill‑Powered Playwright: Evolving UI Automation with Natural‑Language Commands

Background: UI automation before AI

In the Selenium/Appium era, UI automation is essentially "tell the computer: find the button, then click it". The author lists typical difficulties: fragile locators (ids/classes change, virtual scrolling, shadow DOM, canvas), complex wait chains, environment setup (driver versions, devices, CI images), limited assertion expressiveness, and high maintenance cost when pages change.

"Running a test once is fine, but keeping it stable long‑term is hard. As test count grows, maintenance cost explodes—not additive, but multiplicative."

Current shape: Skill‑driven + natural language + Playwright

Skill (skill package) records who tests, what to test, how to ask, and how to translate into code/commands, producing reusable specifications and scripts. Users trigger tests with natural language; the editor’s Agent, constrained by the Skill, selects Web/H5/native, creates executable steps and assertions, and writes them into real repository Page Objects, configuration, and specs instead of one‑off scripts.

Playwright responsibilities

Skill : interaction flow, template dialogues, test case modules, scaffolding and validation.

Playwright : browser automation, project structure, CI integration, trace and HTML reporting.

Compared with "record‑only" or "model‑writes‑script" approaches, Skill pre‑defines engineering constraints (layering, naming, login state, multi‑device switches) to reduce unmaintainable ad‑hoc code.

Practical case: the “Edit” button story

Problem : the "Edit" control looks like a button but is actually an <a class="ant-btn"> link. Traditional workflow required trial‑and‑error with getByRole('button'), consulting Ant Design docs, and guessing version differences, taking about half an hour.

Skill + Playwright solution : the Agent inspects the DOM, tells the user "this is a link button, use getByRole('link', {name: '编辑'}) ", writes the code into the Page layer, and makes it reusable for similar buttons.

Other pitfalls:

Drawer misidentified as modal – Agent corrects the selector to .ant-drawer-title.

Scattered configuration (BASE_URL, login state) – Skill guides the user to set a unified .env and src/config, so changing one variable switches environments.

Separate Web/H5 scripts – a single TEST_TARGET=web flag replaces duplicated conditional code.

Application scenarios & efficiency summary

Typical scenarios include login flow, form submission, popup interaction, list operations, search verification, and multi‑device adaptation.

Measured efficiency improvements:

First‑time authoring: 30‑60 min → 5‑10 min (≈ 80 % reduction).

Debugging/locating: 30‑60 min → 5‑10 min (≈ 85 % reduction).

Maintenance/fixing: 2‑4 h → 10‑30 min (≈ 90 % reduction).

Collaboration barrier: high → low; PM/QA can now participate.

Code reuse across platforms: two codebases → one (≈ 50 % less code).

Failure post‑mortem: 30‑60 min → 5‑10 min (≈ 85 % reduction).

Future development direction

Playwright’s current element locating still relies on manual selectors, which break when the DOM changes. The roadmap evaluates introducing an agent‑browser‑use class that combines visual recognition and semantic understanding so that an LLM can automatically locate elements, making selectors resilient to minor UI changes.

Beyond testing, the vision expands to full‑scene browser automation: crawling, data entry, account management, and RPA, essentially becoming the "first UI‑automation solution for the AI era".

Conclusion

The transformation is not merely fewer lines of code but a systematic shift: scattered troubleshooting experience is captured in Skills/Page objects, enabling instant resolution of similar issues, lowering the entry barrier for non‑developers, and providing reliable, traceable test runs.