RobotAgentBench: Benchmarking Open-World Robot Autonomy Beyond Preset Tasks

Researchers from Aether AI and UC San Diego introduce RobotAgentBench (RAB), a comprehensive benchmark for open-world robot autonomy. The benchmark moves beyond executing explicit instructions to assessing whether robots can autonomously advance a series of tasks from high-level user requests, such as preparing for a guest visit.

Five Complementary Evaluation Dimensions

RAB decomposes autonomy into five dimensions:

Manipulation : converting explicit commands into correct physical actions.

Task Discovery : proactively observing the environment to identify needed tasks.

Situational Reasoning : combining context and user intent to decide how tasks should be done.

Failure Recovery : detecting execution failures and taking remedial action.

Robustness : adapting to unexpected environmental changes and disturbances.

Task-level evaluation covers 90 atomic tasks, 18 short-horizon tasks, and 8 long-horizon tasks (e.g., making coffee, tidying items, heating food). Scene-level evaluation adds four open-ended scenarios: preparing dinner, hosting guests, getting ready to go out, and tidying a room.

Real-to-Sim Pipeline and 50 Interactive Layouts

To preserve physical interaction complexity while enabling scalable experiments, the team developed an agent-assisted, human-in-the-loop real-to-sim pipeline. Starting from a real home, they reconstructed the scene in simulation and expanded it into 50 diverse household layouts varying in room structure, decor, furniture, object appearance, and lighting. This allows testing whether robots generalize when familiar objects move, layouts change, or visual conditions differ.

Unified Plan-Execute-Verify Framework Comparing Five Agent Designs

RAB adopts a unified planner-executor-verifier framework to compare system designs:

VLA/WAM policies augmented with a code verifier that checks results and applies local corrections.

Code planner generating sub-task instructions for learned policies.

Fully code-based planning, execution, and verification.

Hybrid execution dynamically choosing between code control and learned policies.

Full-code system with Recursive Self-Improvement (RSI) that iteratively refines reusable control functions from experience.

Hybrid execution example: code moves the gripper near an object, then a learned policy handles the contact-sensitive grasp. RSI lets agents improve control tools across trials and retain improvements.

The benchmark evaluates 8 VLA models and 3 WAMs, plus coding agents Codex, Claude Code, and Antigravity CLI.

Manipulation vs. Autonomy Gap

Among standalone policies, InternVLA-A1.5 achieves the highest task-level average score (18.0) and excels at failure recovery and robustness. However, these strengths do not translate to open-scene autonomy. Standalone VLA/WAM policies score below 2 on task discovery; open-scene composite scores peak at 2.0. Behavior analysis shows InternVLA-A1.5 and OpenWAM-Alpha spend 88.3% and 84.6% of time stalled — idle or making no effective progress.

Adding coding agents markedly improves task discovery and situational reasoning. Antigravity CLI in a full-code configuration reaches a task-level average of 39.7 and situational reasoning score of 80.0. The best configuration pairs Codex with InternVLA-A1.5 in hybrid execution: task-level average 48.5, open-scene composite 15.4 (vs. Codex full-code at 31.0 and 8.4). Further metrics reveal distinct bottlenecks: InternVLA-A1.5 fails to initiate tasks, while Codex repeatedly retries the same sub-task until time expires.

Planner-Executor Instruction Alignment Matters

Combining different planners and executors shows Antigravity CLI as planner yields the largest gain, while InternVLA-A1.5 is the best executor. Analysis of generated instructions reveals Antigravity CLI uses shorter, simpler instructions with fewer switches and less low-level detail; Codex often produces detailed instructions specifying even which hand to use. This suggests an instruction alignment problem: richer planner descriptions may exceed what learned policies can reliably interpret and execute.

Recursive Self-Improvement (RSI) Delivers Continuous Gains

Over 10 consecutive RSI rounds, all three full-code agents improve on both task-level and scene-level metrics. Agents learn to decompose difficult reaches into feasible intermediate waypoints. For Codex, task-level average rises from 31.0 to 42.3; open-scene composite from 8.4 to 10.8. Stalled time drops from 33.7% to 12.3%; failure recovery rises from 15.4 to 38.9. However, open-scene performance remains limited, indicating further research needed on converting self-improvement into broader, more reliable autonomy.

Sim-to-Real Gap Persists

Real-world evaluation on a wheeled humanoid robot in a matched subset of the simulated environments shows consistent score drops. Codex task-level average falls from 34.2 (sim) to 31.9 (real); open-scene composite from 8.7 to 6.0. InternVLA-A1.5 drops from 17.8 to 13.5. Relative rankings stay similar, supporting simulation's reference value, but richer physical interactions and real-world conditions amplify existing system weaknesses.

The team's prior work Video2World explores whether coding agents can reconstruct executable simulation scenes from a single demonstration video, complementing RAB's focus on autonomy evaluation.

Project page: https://aetherlabsai.github.io/RobotAgentBench/ | Code: https://github.com/AetherLabsAI/RobotAgentBench | Model: https://huggingface.co/AetherLabs-AI/InternVLA-A1.5-RAB | Paper: https://arxiv.org/abs/2610.34649