Introducing UI‑TARS: An Open‑Source Model for Automated UI Interaction

What is UI‑TARS?

UI‑TARS is a native GUI‑agent model that accepts a screen‑capture and a natural‑language instruction, then predicts the next operation needed to fulfill the command. The name references the "TARS" robot from the film Interstellar , implying high autonomy.

Origin: Midscene.js and General‑Purpose Large Models

Midscene.js, open‑sourced by the Web Infra team in 2024, originally leveraged large multimodal models such as GPT‑4V (Sept 2023) and GPT‑4o (May 2024) to enable web UI automation and testing. The project provides three core APIs:

aiAction : drives a large model to perform a series of actions that approximate a human goal.

aiQuery : extracts structured information from a page via natural language.

aiAssert : checks whether the page satisfies a given condition.

A browser plugin also offers a zero‑code experience.

Bottlenecks of Using General‑Purpose Models for UI Automation

1. Need for Engineering to Extract Coordinates

Precise element coordinates are required for execution, but generic models lack fine‑grained numeric understanding. Midscene.js therefore uses JavaScript to extract element types and coordinates, annotates screenshots, and sends the annotated image plus element description to the model, which returns the target element ID. This workaround avoids numeric interpretation but struggles with complex CSS hierarchies and canvas scenes.

2. High Token Consumption

Sending both images and textual element descriptions consumes many tokens, increasing inference cost and latency.

3. Data‑Security Risks

Most developers must call commercial LLM services from Midscene.js, which can be a barrier for internal systems that cannot expose backend data.

4. Unstable Target‑Driven Planning

General models often fail to understand high‑level goals without detailed step‑by‑step instructions, placing extra burden on developers. For example, the brief command “order a sugar‑free milk tea” is less reliable than the detailed sequence “open Jasmine milk tea, click sugar‑free, scroll down, add to cart…”.

UI‑TARS Model – Native GUI Agent

UI‑TARS is trained on a multimodal language model specifically for intelligent UI interaction. It outperforms generic models in the GUI‑agent domain.

The model incorporates four capabilities:

Perception : understands and describes the content of a screenshot.

Action : unifies interaction events to support complex automation tasks.

System‑2 Reasoning : enables reflective thinking to improve task accuracy.

Learning from Prior Experience : uses long‑term memory and dynamic learning to mitigate scarce GUI‑operation data.

Advantages in UI Automation

Shifts from step‑driven to goal‑driven workflows—only the target goal is needed.

Provides reflection and error‑correction capabilities.

Reduces token usage by transmitting only images, speeding up inference.

Eliminates the need for explicit element extraction, handling canvas and desktop scenarios.

Open‑source 7B and 72B parameter GUI‑specialized models (a 2B variant is forthcoming), improving speed and data privacy.

Limitations

UI‑TARS does not achieve 100 % success on GUI tasks. Accuracy drops when a task exceeds twelve steps, and the model still struggles with critical decision points and handing control back to humans.

Integrating UI‑TARS into Midscene.js

Full configuration guide: https://midscenejs.com/zh/choose-a-model

Prerequisites:

Install the latest Midscene Chrome extension or Midscene.js SDK ≥ v0.10.

Deploy a UI‑TARS inference service (see https://github.com/bytedance/UI‑TARS).

Add an environment variable.

MIDSCENE_USE_VLM_UI_TARS=1

Using the Browser Plugin

Install the Midscene Chrome plugin.

Configure the environment variables:

OPENAI_BASE_URL=''  # URL of your inference service
OPENAI_API_KEY=''   # Key for the service
MIDSCENE_USE_VLM_UI_TARS=1  # Enable UI‑TARS model

Code Integration

For complex tasks, use Midscene’s YAML scripts and JavaScript SDK, which also provide a reporting UI for debugging AI execution.

Declarative GUI control via YAML.

Reflections on UI Automation

Do we need gigantic multimodal models?

The UI‑TARS experience shows that smaller, specialized models can match or exceed the performance of massive generic models while offering faster inference (0.5 s–2 s per web page) and lower cost.

How trustworthy is AI for critical tasks?

For high‑risk steps, a “human‑in‑the‑loop” approach is recommended, where AI handles routine selections (e.g., product filtering) but a human confirms final actions such as payment.

References

UI‑TARS repository: https://github.com/bytedance/UI‑TARS

Midscene.js repository: https://github.com/web-infra-dev/midscene