Mobile-Agent-v3 and GUI-Owl: Open‑Source Breakthrough in GUI Automation

Mobile‑Agent‑v3 and GUI‑Owl Overview

Alibaba Tongyi Lab released the Mobile‑Agent‑v3 framework whose core is GUI‑Owl, an end‑to‑end multimodal GUI agent built on Qwen2.5‑VL and fine‑tuned with massive, diverse GUI interaction data. GUI‑Owl unifies UI perception, element grounding, complex reasoning, task planning and action execution in a single policy network.

Explicit Reasoning and Action Translation

Before issuing any concrete command, GUI‑Owl performs an explicit reasoning step, generating a concise “conclusion” to summarise the current step and storing it in the history to keep long‑term interaction efficient. The abstract actions are then translated into device‑specific commands such as ADB for Android or pyautogui scripts for desktop environments.

Innovation 1: Self‑Evolving GUI Trajectory Production

Traditional GUI data collection relies on costly manual annotation. Mobile‑Agent‑v3 builds a large‑scale cloud‑based environment infrastructure (cloud phones and VMs on Alibaba Cloud) that supports Android, Ubuntu, macOS and Windows. Within this infrastructure a four‑stage pipeline generates high‑quality interaction data:

High‑quality Query Generation : automatically creates diverse, challenging queries that cover multi‑step app tasks, cross‑app workflows and complex logical conditions.

Model Rollouts : GUI‑Owl executes the queries in virtual environments, recording screenshots and actions to form interaction trajectories.

Rigorous Correctness Judgment : an evaluation module filters trajectories for correctness, efficiency and alignment with expected behavior; only high‑quality data enter the training set.

Query‑specific Guidance Generation : successful trajectories are used to produce action descriptions, quality‑controlled refinements and synthesized guidance that help the model recover from failures.

This self‑evolution creates a positive feedback loop where generated data improve the model, which in turn produces higher‑quality data, dramatically reducing reliance on human labeling.

Innovation 2: Diverse Foundational Agent Capabilities

GUI‑Owl is equipped with three core abilities—grounding, image captioning and planning—trained on dedicated datasets mixed with general instruction data. The model demonstrates zero‑shot GUI Q&A and strong instruction‑following on unseen tasks. Additional pipelines further enhance specific skills:

UI Element Grounding : precise detection and bounding‑box localization of any UI component, including fine‑grained word/character positioning.

Task Planning : decomposes long‑term goals into ordered sub‑steps, learning from programmatic knowledge extracted from successful trajectories.

Action Semantics : learns causal links between actions and visual state changes, building an internal world model for deeper reasoning.

Innovation 3: Scalable Environment Reinforcement Learning with TRPO

To boost real‑world performance, Mobile‑Agent‑v3 introduces a scalable RL training loop that decouples experience generation from policy updates, enabling fully asynchronous training. The core algorithm, Trajectory‑aware Relative Policy Optimization (TRPO), optimizes a clipped objective that uses trajectory‑level rewards and a replay buffer of successful trajectories, improving stability and sample efficiency for long‑horizon GUI tasks.

Trajectory‑level Rewards : distributes the total reward of a trajectory across each step, encouraging long‑term success.

Replay Buffer : stores high‑quality trajectories and samples them randomly to reduce variance.

Policy Optimization Target : scales loss by the number of steps in the original trajectory to balance updates for high‑resolution screenshots.

Multi‑Agent Collaboration in Mobile‑Agent‑v3

The framework consists of four specialized agents that cooperate in a loop:

Manager Agent (M) : strategic planner that decomposes user instructions into ordered sub‑goals, enriches them with external knowledge via RAG, and dynamically updates the plan based on feedback.

Worker Agent (W) : tactical executor that selects and performs the most relevant sub‑goal, generating a thought, an action command and a step summary.

Reflector Agent (R) : self‑correction module that evaluates the result of each action, classifies outcomes (success, neutral, harmful) and provides causal feedback.

Notetaker Agent (C) : persistent context keeper that records key UI elements and task progress when the Reflector deems the step successful or neutral, supplying memory for future planning.

This loop—plan → execute → reflect → note → re‑plan—enables handling of complex, long‑duration tasks that single agents cannot solve.

Performance Evaluation

GUI‑Owl was benchmarked on several open‑source GUI automation suites:

AndroidWorld : GUI‑Owl‑7B achieved 66.4% success, surpassing same‑size open‑source models; combined with Mobile‑Agent‑v3 the rate rises to 73.3%.

OSWorld : 29.4% success for GUI‑Owl‑7B, improving to 37.7% when integrated with the multi‑agent framework.

MMBench‑GUI : GUI‑Owl‑32B outperforms GPT‑4o and Claude 3.7, setting a new SOTA for open‑source GUI agents.

AndroidControl : GUI‑Owl‑32B leads among open‑source models, confirming its strength on mobile GUI tasks.

Ablation Studies

Key components validated through ablations include:

TRPO policy: raises OSWorld‑Verified success from 27.1% to 34.9%.

Online filtering, replay buffer and experience management: essential for training stability and efficiency.

Number of historical images and interaction‑step budget: more context correlates with higher accuracy.

Reasoning data synthesis (offline hint‑guided rejection sampling, multi‑agent distillation, iterative online rejection sampling): progressively enhances logical reasoning capabilities.

Conclusion and Outlook

Mobile‑Agent‑v3 and its core GUI‑Owl model advance the state of GUI automation by providing a scalable environment infrastructure, diverse foundational capabilities, and an extensible reinforcement‑learning loop. The self‑evolving trajectory pipeline addresses the long‑standing data scarcity problem, while the multi‑agent architecture demonstrates the potential of collaborative agents for complex, adaptive GUI tasks.