Ant Group OpAgent: Online RL‑Powered Open‑Domain Browser Automation Agent

Task Definition and Baseline Challenges

Automated website navigation requires a user query and an initial URL; the model must interact with the page through a sequence of actions such as clicking a search box, entering text, submitting, sorting, and opening a target video. Although the goal is clear, real‑world pages present three major obstacles:

Perception difficulty: complex DOM trees and ambiguous definitions of “interactive” elements make it hard for a language model to distinguish clickable controls from layout containers.

Timeliness gap: offline‑collected navigation trajectories quickly become stale as web interfaces evolve, causing supervised fine‑tuned (SFT) models to degrade when deployed.

Implicit interaction logic: certain actions (e.g., typing pinyin instead of Chinese characters) succeed only after trial‑and‑error, a behavior not captured in static datasets.

Three‑Stage Technical Roadmap

To address these issues the team designed a “three‑stage” pipeline:

Multi‑Task Supervised Fine‑Tuning (MT‑SFT) equips the visual‑language model (VLM) with basic web‑operation capabilities.

Online Reinforcement Learning (RL in the Wild) continuously adapts the policy through real‑time interaction with live websites.

Four‑Module Agent Framework – Planner, Grounder, Reflector, Summarizer – handles long‑horizon tasks by separating perception, grounding, execution, monitoring, and answer synthesis.

MT‑SFT Details

The VLM learns three ability dimensions: Planning (predicting page changes after an action), Acting (choosing click, type, or scroll), and Grounding (outputting pixel coordinates). Corresponding datasets are:

Planning – WebDreamer (million‑scale).

Grounding – automatically crawled coordinate annotations (million‑scale).

Acting – manually labeled “what to do on the current page” (only a few thousand samples).

Because the Acting set is tiny, loss weights are dynamically scaled by dataset size so that gradients from the small set are amplified, keeping training intensity balanced across the three dimensions.

Online RL Infrastructure and Reward Design

The RL system is built on four layers:

VLM decision layer that outputs a JSON action with type and pixel coordinates.

Playwright execution layer that translates the JSON into real browser commands.

Ray‑based distributed browser cluster for multi‑GPU, multi‑node parallel sampling.

Execution environment on ECS hosting WebArena mirror sites (GitLab, Reddit, etc.). Each interaction round generates a trajectory; rewards are provided by two complementary mechanisms:

WebJudge (heavy reward) scores the full trajectory on task completion, action effectiveness, and efficiency (1–5 each).

Decision‑tree reward (light reward) penalizes ineffective clicks by checking URL change, DOM hit testing, and image similarity, falling back to the language model when needed.

The combined reward encourages both immediate valid actions and overall task success, leading to steadily rising reward curves during training.

Four‑Module Agent Framework

After the RL stage, the agent operates with the following modules:

Planner : decomposes the user query into sub‑tasks and issues textual instructions to the Grounder.

Grounder : converts Planner’s text into concrete pixel‑level actions using the VLM.

Reflector : monitors page state, decides whether the task is complete, needs continuation, or is stuck in a loop.

Summarizer : aggregates the full interaction history and generates the final answer.

Only the Grounder handles pixel coordinates; the other three modules remain pure text reasoning and can be powered by general‑purpose models such as GPT‑4 or Qwen, reducing the learning burden on the end‑to‑end system.

Evaluation Results and Current Status

On the GUIAct benchmark, MT‑SFT improves Qwen2.5‑VL‑72B over GUI‑Actor and UI‑TARS on Web‑Multi (EM and Step SR). Online RL further raises scores: on a private test set the average score rises from 2.0 (baseline) → 3.0 (RL only) → 3.5 (SFT + RL). On WebArena, RL boosts Overall from 27.4 % to 38.1 % and Pass@5 by +11 pp (Pass@1 +8 pp). After integrating the four‑module agent, the single‑model Pass@1 reaches 71.6 % in the January 2026 WebArena leaderboard.

Open‑Source Release

The code, models, and a demo are publicly available on GitHub, Hugging Face, and ModelScope, and the full technical report is posted on arXiv (2602.13559).

Future Directions

The team is exploring joint multi‑task training (JAPO) that optimizes Planner and Grounder together within a unified RL loop, with early promising results.