Workflow Gym: Moving Beyond Simulated Tests to Bridge the Gap for GUI Agents

Workflow‑GYM: Towards Long‑Horizon Evaluation of Computer‑use Agentic tasks in Real‑World Professional Fields (paper on HuggingFace https://huggingface.co/papers/2606.11042 and arXiv https://arxiv.org/abs/2606.11042) evaluates GUI agents on authentic professional software instead of toy benchmarks.

From Toy Benchmarks to Real‑World Tasks

Earlier benchmarks (OSWorld, PC‑Eval, AmbiBench, GUI‑360) cover 8–25 everyday applications (Chrome, VS Code, LibreOffice, Word, Excel, etc.) and report success rates above 70 %. Tasks are short, low‑complexity, and can be solved by memorising patterns.

Workflow‑GYM expands the arena to 56 professional tools (FreeCAD, QGIS, KiCad, Blender, OBS, etc.) and 338 curated workflows. Each workflow averages over 100 steps; the longest contains 110 steps. Tasks are extracted from domain‑expert workflows, filtered from >1,000 candidates to ensure no publicly available tutorial solves them, and packaged as VM images (Windows and Ubuntu) with pre‑installed, pre‑configured software.

Four Core Design Principles

Realism : Tasks are mined from real expert workflows; 1,000 candidates were narrowed to 338 after rigorous filtering.

Professionalism : Coverage spans six top‑level domains and 23 sub‑domains, including engineering design, scientific computing, multimedia creation, GIS, data analysis, finance, bio‑informatics, chemistry modeling, and game engines.

Long‑Horizon : Difficulty stratified by step count – Easy (30‑44 steps, 38.2 % of tasks), Medium (45‑60 steps, 47 %), Hard (≥61 steps, 14.8 %). Interaction rounds per task often reach 80‑400, far beyond the nominal step count.

Verifiability : Each task has a deterministic scoring rule – either binary file comparison or VLM‑based screenshot verification. Human‑vs‑machine agreement on a 60‑task sample is 98.3 %.

Experimental Setup

Three state‑of‑the‑art agents – Gemini 3.1 Pro, Gemini‑3‑Flash, and Kimi‑K2.6 – were evaluated without step‑by‑step hints. Each task allowed up to 400 interaction rounds and was run three independent times. Metrics: pass@3 (success in any of three runs) and single‑run success rate.

Key Findings

Gemini 3.1 Pro single‑run success rate: 30.67 % (fails two out of three attempts on professional GUI tasks).

Gemini‑3‑Flash pass@3 on Hard tasks: 2.67 % (essentially unsolvable for the lightweight model on the 50 hardest tasks).

Kimi‑K2.6 pass@3 on Hard tasks: 25.33 % vs. Gemini 3.1 Pro 21.33 % – raw model size does not guarantee long‑horizon robustness.

Domain‑wise performance: Kimi excels in data analysis and multimedia creation; Gemini is stronger in GIS, engineering design, and scientific simulation.

Failure Analysis

Failures are categorized at two levels.

Result‑level

Workflow incompletion – the agent stops before finishing the sequence.

Execution inaccuracy – the workflow finishes but the final output is incorrect.

Stronger models reduce incompletion rates, but execution inaccuracies remain prevalent.

Process‑level pitfalls (observed across all agents)

Long‑term consistency break : loss of earlier parameters or context leads to cascading errors.

Error propagation without self‑repair : after a mistake the agent continues operating instead of undoing or correcting.

Goal drift : final output misses critical acceptance criteria (e.g., forgetting to export a video).

Insufficient understanding of professional UI : lack of domain‑specific knowledge causes random clicks on unfamiliar panels.

These stem from a structural mismatch: human GUI use follows a continuous perception‑action loop, whereas current agents operate in a discrete “screenshot → inference → click” cycle, which degrades on intricate professional interfaces.

Implications

Advancing GUI agents requires addressing long‑term consistency, error recovery, and domain knowledge injection – challenges not solved by scaling model parameters alone.

Future Directions

Planned extensions include adding more software, increasing instruction diversity, integrating evaluation and training environments for RL‑style closed‑loop learning, and exploring conversion of GUI trajectories into standard tool‑chain interfaces.

Overall, the benchmark shows that current GUI agents achieve only ~30 % single‑run success on authentic professional workflows, highlighting a substantial capability gap between laboratory benchmarks and real‑world professional tasks.