Introducing UI‑TARS: A Native GUI Agent Model Integrated with Midscene.js for Multimodal UI Automation

Origin: Midscene.js Uses General‑Purpose Large Models for UI Automation

OpenAI released GPT‑4V in September 2023 and GPT‑4o in May 2024, expanding multimodal capabilities that make it feasible to use large models for UI automation. Midscene.js was created to provide APIs such as aiAction , aiQuery , and aiAssert for natural‑language‑driven interactions.

Principles and Bottlenecks of Using General Models for UI

Bottleneck 1: Need for Engineering to Extract Coordinates

Accurate UI actions require element coordinates, which generic models lack; Midscene.js extracts coordinates via JavaScript, annotates screenshots, and sends them to the model for element ID prediction, though CSS and canvas elements remain challenging.

Bottleneck 2: High Token Consumption

Sending both images and element descriptions consumes many tokens, increasing cost and latency.

Bottleneck 3: Data Security Risks

Commercial model services often require sending internal data externally, which many enterprises cannot permit.

Bottleneck 4: Unstable Target‑Driven Planning

Models struggle with sparse instructions; detailed step‑by‑step prompts improve stability but increase developer burden.

UI‑TARS Model – Native GUI Agent

UI‑TARS is a multimodal language model fine‑tuned for intelligent UI interaction, outperforming generic models in the GUI agent domain.

It incorporates human instructions, screenshots, and previous actions into its self‑attention mechanism to reason about the next operation.

Perception : Understands and describes screenshot content.

Action : Executes diverse interaction events.

System‑2 Reasoning : Performs reflective reasoning to improve task accuracy.

Learning from Prior Experience : Uses long‑term memory to mitigate data scarcity.

Advantages of UI‑TARS in UI Automation

Shifts from step‑driven to goal‑driven automation.

Provides reflection and error‑correction capabilities.

Reduces token usage by transmitting only images.

Handles canvas and desktop scenarios without element extraction.

Open‑source 7B and 72B parameter models improve speed and privacy.

Limitations

Execution accuracy is not 100%; tasks with more than 12 steps may cause mispredictions, and the model’s ability to hand over control to humans needs improvement.

Integrating UI‑TARS into Midscene.js

Full configuration guide: https://midscenejs.com/zh/choose-a-model

Prerequisites:

Install the latest Midscene Chrome extension or SDK (v0.10+).

Deploy a UI‑TARS inference service (see https://github.com/bytedance/UI‑TARS).

Add an environment variable:

MIDSCENE_USE_VLM_UI_TARS=1

Using the Midscene.js Browser Plugin

Install the Midscene Chrome plugin.

Configure environment variables:

OPENAI_BASE_URL=''  # URL of your inference service
OPENAI_API_KEY=''   # API key for the service
MIDSCENE_USE_VLM_UI_TARS=1  # Enable UI‑TARS model

Code Integration

For complex tasks, use Midscene’s YAML scripts and JavaScript SDK, which provide declarative GUI control and detailed AI execution reports.

Supports Playwright and Puppeteer for end‑to‑end testing.

Reflections on UI Automation

Do We Need Extremely Large Multimodal Models?

UI‑TARS shows that smaller, specialized models can achieve comparable speed (0.5‑2 s per inference) and accuracy without the overhead of massive parameters.

Trustworthiness of AI for Critical Tasks

AI should assist rather than replace humans in high‑risk decisions, employing a Human‑in‑the‑Loop approach for tasks like final payment confirmation.

References

UI‑TARS repository: https://github.com/bytedance/UI‑TARS

Midscene.js repository: https://github.com/web‑infra‑dev/midscene