Why Top AI Agents Fail in Real Work: Inside the Trainee‑Bench Benchmark

Prologue: The Mojave Desert and the Birth of Autonomous Driving

In 2005 the DARPA Grand Challenge took place in Nevada’s Mojave Desert, where autonomous vehicles had to navigate hundreds of kilometres without high‑definition maps or human remote control. The victory of the vehicle "Stanley" demonstrated that machine intelligence is valuable only when it can survive and complete tasks in an unknown physical world.

The Execution Gap: High Scores, Low Capability

Current agent benchmarks are "oracle‑based"—they provide perfect context and expect perfect answers—whereas real workplaces are "partially observable" with ambiguous requirements, sudden interruptions, and hidden information. This mismatch creates an execution gap: agents that score near‑perfect on static tests become "giant infants" that need constant human supervision in production.

Trainee‑Bench: Simulating a Real Workplace

The research teams from Fudan, Shanghai AI Lab, Zhejiang University and others built a highly realistic "workplace simulator" called Trainee‑Bench . It evaluates agents on the first day of a virtual job, without any "god‑view" assistance, forcing them to rely on their own perception, exploration, and scheduling abilities. The benchmark assesses agents across three technical dimensions.

Dimension 1: From Linear Reasoning to Dynamic Scheduling

Priority judgment: Can the agent distinguish urgent from routine tasks?

Suspend and resume: After handling an emergency, can it return to the previous progress without loss?

Dimension 2: From Full‑Map to Active Exploration

In a "mapless" environment the agent receives no file locations or tool instructions. It must explore using commands such as ls and grep to discover directories, read documents, and gradually construct a cognitive map of the workspace.

Dimension 3: From One‑Shot Completion to Continuous Learning

Trainee‑Bench evaluates agents over two consecutive days. After Day 1 the agent receives feedback and is expected to improve on Day 2. Surprisingly, most agents performed worse (score dropped from 0.42 to 0.36), indicating that current large‑model “experience” is shallow or over‑fitted and harms performance in a changing environment.

Construction Steps of Trainee‑Bench

Meta‑Task design: To prevent memorization, 181 meta‑task rules are generated with random seeds, creating diverse NPC personalities, file structures, and hidden clues that require active exploration.

Dynamic composite scenes: Multiple independent tasks are interleaved on a timeline, each with its own priority and possible dependencies, testing multi‑task planning.

Automatic verification: Embedded checkpoints automatically assess each step, providing fine‑grained natural‑language feedback rather than only final outcomes.

Evaluation Results: Top Models Stumble

Seven state‑of‑the‑art models—including Gemini‑3‑Flash, GPT‑5.1, GPT‑4o, Claude‑4‑Sonnet—were tested. The best success rate was only 35% (Gemini‑3‑Flash). When concurrent tasks increased from 2 to 6, most models showed a sharp performance drop, confirming that multi‑threaded scheduling remains a critical weakness. The continuous‑learning test further revealed that experience reuse can degrade performance.

Re‑defining Agent Value: Equivalent Human Time

{Value} = {Human autonomous time} – {Agent time} + {Human supervision time}

If an agent requires frequent human correction, the metric becomes negative, meaning the agent wastes compute rather than delivering productivity. Only agents that achieve zero human hand‑over across exploration, scheduling, and learning truly possess sustainable commercial value.

Conclusion: Seeking the “Stanley” of Digital Workplaces

The paper argues that the AI community should shift focus from merely scaling model parameters to fostering autonomous learning in agents. Agents that can independently handle complex, dynamic tasks, require minimal human guidance, and survive in mapless environments will earn a genuine “workplace badge” in the future.