EMERGE-Policy: Multi-Agent Framework Unifies VLA, World Models for Embodied AI

In long-standing robot manipulation research, single models such as Vision-Language-Action (VLA) models have been widely studied but struggle with complex long-horizon tasks and unexpected disturbances. To address this, a team led by Professor Li Xiu at Tsinghua University's Shenzhen International Graduate School, including master's students Fang Zhirui, Yu Chi, and Chen Ziyang, in collaboration with multiple universities, introduced EMERGE-Policy — a novel multi-agent asynchronous concurrent framework.

Breaking the Single-Model Paradigm

EMERGE-Policy moves beyond the "single policy decides everything" paradigm. It integrates heterogeneous models — VLA, Whole-body Action Models (WAM), world models, and verifiers — into a unified skill library that the system can call as tools. System-level intelligence emerges from the closed-loop collaboration among a Main Agent, role-specific Sub-Agents, and this skill library.

"Thinking outside the brain means skillfully engaging entities external to our heads." — Annie Murphy Paul, The Extended Mind

The architecture mirrors human cognition: perception (eyes), manipulation skills (hands), imagination (brain), and verification (feedback) are handled by distinct, specialized components.

Heterogeneous Skill Library

The Unified Skill Library encapsulates three categories of skills as standardized tool interfaces (Figure 1):

Operational Skills: VLA (e.g., π_0.5) for high-frequency, precise end-effector trajectory and physical control; motion primitives for coarse-grained or large-range motion planning.

Imagination & Prediction Skills: World Models (e.g., Cosmos Policy) act as simulators, predicting future video states before action execution — enabling "think before you act."

Evaluation & Verification Skills: Verifier models check preconditions before execution, completion criteria after execution, and score candidate trajectories generated by the world model during mental simulation.

Combining imagination and evaluation skills allows the system to consider "posterior" outcomes for better task completion (Figure 2).

System Architecture: Graph-Based Scheduling

Main Agent: Decision and Scheduling Hub

The Main Agent maintains global task state, decomposes long-horizon tasks into subgoals, and uses Hierarchical Context Engineering to shield low-level perceptual redundancy, invoking Sub-Agents and tools via structured interfaces (Figure 3).

Role-Specific Sub-Agents: Independent Specialist Units

Sub-Agents run in isolated contexts to handle high-density proprietary data:

Perception Sub-Agent: Estimates 3D bounding boxes and spatial semantics.

Verification Sub-Agent: Calls verifier tools in real time to assess goal completion.

Monitor Sub-Agent: Watches for physical anomalies during execution.

Three-Layer External Memory and Context Management

To prevent memory loss in long-horizon tasks, the system employs three file-based memory layers (Figure 4):

PLAN.md: Records the task graph and current progress.

HISTORY.md: Appends historical observations and tool-call summaries.

MEMORY.md: Dynamically updates and corrects environment facts.

When context limits are reached, a Memory Integration process automatically compresses history, ensuring stable operation over ultra-long sequences.

Experimental Results: Comprehensive Benchmark Gains

EMERGE-Policy was evaluated on four widely adopted benchmarks and real robots:

Standard LIBERO: Using Cosmos Policy as a world-model skill for trajectory imagination and action selection, EMERGE-Policy achieved 99.2% average success rate (+0.7% over the base model). With π_0.5 as the execution skill, success reached 98.8% (+2.0% over base).

LIBERO-Plus (perturbation test): Under lighting changes and camera pose shifts, EMERGE-Policy (with world model) scored 93.9% completion, a 11.7% improvement over the standalone Cosmos Policy baseline.

RoboDojo-Sim: On the Memory dimension: 25.00 score / 22.51% success ; on the Open dimension: 19.98 score / 11.20% success — far exceeding comparable models, indicating strong historical information encoding and non-Markovian reasoning.

Real-World Deployment: Multi-Layer Cup Stacking

The team deployed EMERGE-Policy on a real manipulator for a high-difficulty "multi-layer cup stacking" task: stacking cups into a 3-2-1 pyramid (Figure 6a).

High stability: In 100 standard real-robot trials, the system showed extremely high completion rates (Figure 6b).

Strong disturbance rejection: Under human sabotage (destroying stacked cups), cup size variations (5–15%), color changes, and camera viewpoint shifts, the system maintained 85–94% success rates via branch-stack recovery and online re-planning with the tool library (Figure 6c).

Conclusion

EMERGE-Policy opens a new research path for embodied intelligence: instead of demanding a single omnipotent model, decoupling VLA, WAM, world models, and verifiers into collaborative tools, orchestrated via a graph-structured multi-agent system, yields a highly adaptive, robust, full-stack embodied intelligence system. The Tsinghua Intelligent Computing Laboratory plans to extend the EMERGE series to further articulate principles for building modern embodied intelligence systems.

References

Paper: EMERGE-Policy: A Robot Mind Emerges Beyond a Single Policy — https://arxiv.org/abs/2608.29896v2 Project website: https://emerge-policy.github.io/EMERGE-Policy/ Code repository: https://github.com/EMERGE-Policy/EMERGE-Policy Lab website:

https://thusigsiclab.github.io/thu.github.io/