REAL: Embodied Agents Navigate Open Worlds Without Oracle Perception or Perfect Instructions

Real-world household tasks often begin with incomplete information: the robot does not know where the target object is, and the user cannot always specify the exact desire in a single instruction. For a vague command like "get me some snacks from the dining table," the robot must actively search for candidates and, upon finding bread and multiple donuts, ask the user which one to retrieve.

Addressing this, researchers from Shanghai Jiao Tong University and Shanghai AI Laboratory propose REAL (accepted at ECCV 2026), an explorable, interactable, and deployable visual embodied agent framework for open-world mobile manipulation. It builds a complete technical pipeline spanning simulation, data generation, model training, system evaluation, and real-robot deployment.

Core Problem: Two Over-Idealized Assumptions

Existing embodied agent simulators rely on two assumptions that break in reality:

Full environment knowledge — simulators provide oracle perception APIs that directly return object counts and positions, but real robots must actively explore to discover objects.

Clear user intent — benchmarks assume precise instructions, yet real users often give vague requests and refine them based on the robot's feedback.

REAL unifies active exploration, visual target grounding, natural-language intent disambiguation, and physical execution into a single decision loop. The agent receives only first-person RGB observations and structural priors (support regions like tabletops, cabinets) from a mapping system. When exploration reveals multiple candidates, the agent proactively reports findings and asks for clarification (e.g., "I found a bread and two different donuts; which one do you want?"), making the query a natural part of the execution flow.

Agentic Architecture: Brain, Tools & Interfaces, State & Memory

Built on the InternUtopia simulation platform, REAL's top-level architecture comprises three components:

Brain : Qwen3-VL-8B-Instruct serves as the backbone vision-language model, processing multimodal observations, maintaining a structured self-summarized history state, and performing high-level task planning and tool calling.

Tools & Interfaces : A standardized toolchain that does not depend on privileged perception:

Four-step exploration tools: Nav_to, Walk_around, Show_object_by_category, Gaze_at.

Low-level control interfaces: Pick, Place, Open, Close.

Active interaction tool: Ask.

The Model Context Protocol (MCP) provides a unified interface standard; the high-level brain emits discrete tool calls executed via an MCP server, decoupling cognitive planning from low-level execution.

State & Memory : MCP tool returns are organized into structured multimodal percepts — visual segmentation masks with object categories, raw RGB observations, user feedback, and the agent's self-summarized sub-goals and history — preserving relevant information while avoiding context explosion.

Three Core Contributions

Privilege-free physical simulation : The simulator removes direct object-localization APIs, forcing the agent to explore freely via the toolchain. A simulated user is introduced so the agent can dynamically query and align intent during execution.

Automated data generation & two-stage training pipeline : Tasks are auto-composed from scenes and assets; a rule-based planner generates expert trajectories, and Gemini 3 Pro annotates step-by-step reasoning and structured history. Stage 1: Supervised Fine-Tuning (SFT) teaches basic tool-use patterns, including rewriting clear instructions into vague ones and injecting Ask calls with simulated user replies. Stage 2: Online Reinforcement Learning with Group Sequence Policy Optimization (GSPO) — a sequence-level importance-ratio clipping method that better matches sequence-level rewards and stabilizes variable-length trajectory RL. Rewards combine environment state changes, failure penalties, and an ask-budget to encourage necessary queries while preventing over-reliance.

Unified-interface sim-to-real deployment : REAL-Bench (241 task instances) validates the framework. Thanks to MCP, the same high-level policy transfers zero-shot to a real dual-arm mobile robot.

Technical Implementation Details

Active Exploration & Interaction Without Privileged Perception

The agent receives only structural priors — support-region IDs and semantic categories (e.g., tabletop, cabinet). To locate targets, it runs the four-step exploration chain (navigate, multi-view scan, category detection, align gaze). Detected 2D segmentation masks are back-projected to 3D to build a temporary exploration map; Set-of-Mark (SoM) assigns visual IDs for subsequent manipulation, enabling visual target grounding and disambiguation of same-category objects with different appearances. A simulated user driven by a large model answers based on discovered objects and task configuration, making active communication an execution step triggered by exploration results.

High-Low Decoupling via MCP

MCP abstracts both simulation and real environments into a consistent interaction space through unified tool interfaces. The high-level VLM focuses on planning and state tracking, emitting discrete tool calls. On the real robot, continuous physical control is handed to specialized skills — a fine-tuned π₀.₅ model handles open, close, pick, place — so the high-level policy transfers zero-shot without retraining.

SFT + GSPO: Tool Alignment First, Then Closed-Loop Decision Making

Qwen3-VL-8B-Instruct balances multimodal capability and inference efficiency. Long-horizon context bloat is mitigated by modeling the decision process as a POMDP with structured self-summarized history, avoiding full-trajectory concatenation at each step.

SFT (Tool Alignment) : Rule-based planner generates expert trajectories; an auto-annotation module rewrites some clear instructions as vague ones and inserts Ask calls with simulated user responses, teaching the base tool-calling paradigm.

Online RL (Adaptive Capability) : GSPO performs closed-loop RL from the 1-epoch SFT checkpoint. Sequence-level clipping improves stability for variable-length trajectories. The reward mixes environment progress, failure penalties, and an ask budget, encouraging the agent to decide autonomously when to explore, replan, or query the user.

REAL-Bench: Evaluation Design & Results

REAL-Bench contains 241 task instances across four families:

Furniture-Distractor Pick-and-Place (FDP) — 72 tasks: cross-container pick-place with same-category furniture distractors, testing active exploration and basic manipulation loop.

Furniture & Object-Distractor Pick-and-Place (FODP) — 56 tasks: both furniture- and object-level distractors in open-vocabulary settings, testing fine-grained visual grounding and disambiguation.

Furniture-Distractor Open/Close (FDO) — 48 tasks: operations on articulated furniture, testing dynamic visual changes and temporal consistency.

Simulator-User-Loop (SUL) — 65 tasks: deliberately vague instructions, testing proactive Ask usage and goal alignment from user feedback.

Together they cover the full explore–ground–manipulate–communicate loop.

Key Experimental Findings

Tool alignment is prerequisite for perception–action loop. Untrained Qwen3-VL-8B scores 0% on FDP, FODP, FDO and 1.5% on SUL. After 1-epoch SFT, FDP jumps to 45.8%.

Pure imitation scaling can hurt open-scene generalization. Increasing SFT to 2 epochs raises FDP to 65.3% but drops FODP from 30.4% to 28.6%, indicating overfitting to expert trajectories.

Online RL restores adaptive exploration. GSPO from the 1-epoch SFT checkpoint lifts FODP to 33.9%, showing environment feedback drives dynamic adjustment rather than trajectory memorization.

Online RL yields largest gains on interactive tasks. REAL-8B achieves 56.9% on SUL , surpassing all zero-shot VLM baselines, including the teacher model Gemini-3-Pro-Preview (53.8%) and GPT-5 (52.3%), demonstrating RL's particular strength for tasks requiring active interaction.

Real-Robot Deployment: 60 Episodes, 78.3% End-to-End Success

The trained high-level policy was zero-shot transferred to an ARX LIFT2 dual-arm mobile robot :

High-level VLM connects directly to the physical backend via the unified MCP tool interface — no retraining.

Low-level manipulation is handled by a fine-tuned π₀.₅ model for open, close, pick, place, letting the agent focus on visual grounding, state tracking, and high-level planning.

Across 60 real-world episodes , the system achieved 78.3% end-to-end success rate with no unrecoverable crashes. Over 600 VLA primitive executions, overall executability was 85.3% .

A representative task — "put the bread in the microwave" — illustrates long-horizon planning and state maintenance. The robot did not go straight to the cabinet for bread; instead, it first opened the microwave, then fetched the bread, returned, placed it inside, and closed the door. This ordering reflects understanding of task preconditions and action dependencies. The structured history state continuously records only decision-relevant, tool-verified progress, avoiding rapid context growth and reducing state forgetting and error accumulation.

Conclusion

REAL redesigns the information boundary between simulation training and real deployment, freeing embodied agents from two traditional idealizations — perfect perception unavailable in reality, and the assumption that users always give complete, unambiguous instructions. Instead, agents must actively gather visual evidence through exploration, fill intent gaps via communication when needed, and finally invoke deployable physical skills to complete tasks. The unified MCP interface deeply decouples high-level visual reasoning from low-level physical execution, enabling the same high-level policy to transfer zero-shot from simulation to a real dual-arm mobile robot. REAL-Bench and real-robot experiments consistently show that teaching models to maintain closed-loop decision-making under "partial observability, ambiguous language, and execution failures" provides a viable path from constrained benchmarks to the real world.