Annotation‑Free Mobile GUI Agents: How MobileForge Builds a Data‑Flywheel

Mobile GUI agents have made rapid progress, but adapting them to the fragmented and constantly evolving mobile app ecosystem remains costly because each new app traditionally requires manually written tasks, expert demonstrations, or labeled reward signals.

MobileForge Overview

The paper MobileForge: Annotation‑Free Adaptation for Mobile GUI Agents with Hierarchical Feedback‑Guided Policy Optimization (arXiv:2606.19930) proposes a closed‑loop system that removes all manual annotation. It consists of two tightly coupled components:

MobileGym : an interaction and evaluation backbone that explores target apps, converts raw exploration traces into executable tasks, and provides fine‑grained, hierarchical feedback.

HiFPO (Hierarchical Feedback‑Guided Policy Optimization): a trainer that turns the multi‑step feedback into step‑level policy‑update signals.

MobileGym: From Real‑App Interaction to Task Generation

MobileGym addresses the data‑source problem. It first performs target‑app exploration , using APK activity metadata and current screenshots to define functional exploration goals and traverses the app in a depth‑first manner. Every state transition records before/after screenshots, the performed action, target UI element, metadata, and a natural‑language summary, forming an evidence pool.

Next, MobileGym‑Curriculum converts each exploration trace into a task represented by a five‑tuple (instruction, step‑budget, core function, variation type, precondition). The curriculum ensures that every generated task is anchored to a real observed behavior.

Finally, MobileGym‑Critic evaluates complete rollouts with a hierarchical evaluator that outputs three feedback types: a trajectory‑level outcome label, step‑level process label, and a corrective hint . The hint summarizes failure reasons, undesirable actions, and suggested alternatives, enabling fine‑grained learning from long‑chain mobile tasks.

HiFPO: Turning Feedback into Policy Updates

HiFPO orchestrates four steps. First, it performs hint‑augmented multi‑attempts : for each task the current policy tries K times; after a failure, the critic‑generated hint is appended to the instruction for the next attempt, allowing the agent to reuse experience within the same task.

Second, task filtering discards tasks that the model already masters (all attempts succeed) while retaining fully failed and partially successful tasks, because the latter still contain valuable local actions.

Third, trajectory and step selection picks the highest‑quality successful trajectory or, if none exist, the failed trajectory with the greatest proportion of reasonable steps, and keeps only the step‑level actions judged correct by the critic.

Fourth, HiFPO applies a hint‑contextualized step‑level GRPO update. Each step‑level sample includes the task, screenshots, interaction history, and the applicable hint; the model samples multiple candidate actions and compares them using a regularized GUI‑action reward.

Experimental Setup

Two benchmarks are used:

AndroidWorld (in‑domain): 116 tasks from the AndroidWorld app ecosystem. Models evaluated are the general‑purpose VLM Qwen3‑VL‑8B and the GUI‑specialized GUI‑Owl‑1.5‑8B.

MobileWorld GUI‑only (cross‑domain): 117 tasks from a different app set, with no MobileWorld data used during training.

MobileForge generated 3,249 candidate tasks from 20 apps (527 source trajectories). Subsets of 200, 400, and 900 tasks were used to study scaling effects.

Results

On AndroidWorld, using only automatically generated data, Qwen3‑VL‑8B improved its Pass@3 from 55.2 % to 67.2 % (900 tasks), approaching the closed‑source GUI‑specific baseline GUI‑Owl‑1.5‑8B at 69.0 %. The adapted ForgeOwl‑8B reached 77.6 % Pass@3, with Pass@1 rising from 56.0 % to 67.2 % and hard‑task success climbing from 19.3 % to 29.8 %.

Cross‑domain evaluation on MobileWorld showed ForgeOwl‑8B achieving 41.0 % success, surpassing the GUI‑Owl‑1.5‑8B baseline (37.6 %) and other open‑data mobile GUI agents. ForgeQwen3‑8B improved from 7.6 % to 10.3 %.

Ablation Studies

Hint impact : without hints, multi‑attempt success rate was 52.0 %; with hints it rose to 77.0 %, and Pass@3 increased from 49.0 % to 72.5 %.

Training target : no‑hint SFT performed worst; hint‑SFT improved modestly; hint‑contextualized GRPO consistently yielded the best results, reaching 50.9 % Pass@1 with 900 tasks.

Task filtering : removing mastered tasks (instead of all failures) gave the highest success‑rate range [0.0, 0.9].

Critic model : using Gemini 2.5 Pro gave the best results, but substituting Qwen3‑VL‑8B still raised Pass@1 from 40.5 % to 44.8 % and Pass@3 from 55.2 % to 60.3 %.

Curriculum grounding : a Broccoli example showed that trajectory‑based curriculum covered a broader set of functions (shopping list, cooking assistant, settings, media sharing) compared with a landing‑screen‑only approach that over‑focused on recipe creation/deletion.

Case Study

In an AndroidWorld task that requires deleting three expense items in the “Pro Expense” app, the baseline Qwen3‑VL‑8B loses the task flow after the first deletion. After MobileForge adaptation, ForgeQwen3‑8B maintains the deletion intent across repeated UI cycles and successfully removes all three items.

Failure‑Rate Reduction Analysis

Tag‑wise analysis shows notable reductions in verification, search, complex UI, screen reading, repetition, and information‑retrieval categories, while tasks involving game‑playing, multi‑app coordination, memorization, and math‑counting remain challenging.

Conclusion

MobileForge delivers a fully annotation‑free data flywheel for mobile GUI agents: the agent explores real apps, MobileGym turns interactions into tasks and hierarchical feedback, and HiFPO converts successes, failures, and corrective hints into step‑level policy updates. This eliminates the need for manual task authoring, expert demonstrations, or reward labeling, enabling continuous self‑improvement of mobile GUI agents in a dynamic app ecosystem.