Why One Model Can’t Win: RoboHarness Orchestrates Heterogeneous Robot Policies

Motivation for Heterogeneous Strategy Orchestration

Recent advances such as VLA, WAM, RL, TAMP, and MPC excel in their own domains, but real‑world tasks demand simultaneous semantic understanding, closed‑loop control, geometric precision, and long‑horizon reasoning—requirements that exceed any single controller’s capability. The authors argue that waiting for a universal embodied model is unrealistic and propose coordinating existing policies instead.

RoboHarness Overview

The system, described in the paper "RoboHarness: Memory‑Driven Orchestration of Heterogeneous Robot Policies for Long‑Horizon Planning" (arXiv:2607.18060), encapsulates diverse controllers (VLA, RL, TAMP, etc.) as callable skills. A coding agent performs high‑level task decomposition and routes sub‑tasks to the most suitable policy. When the state distribution of the next policy is out‑of‑distribution, RoboHarness generates a bridging trajectory to ensure a smooth handoff. Integration does not require shared action spaces, joint training, or data exchange, allowing painless addition of new strategies such as WAM, VLN, or MPC.

Three Structured Auxiliary Skills

Understanding Skills : Analyze raw inputs to extract richer information (semantic similarity, pose uncertainty, exposure) that defines each policy’s capability boundary.

Memory Skills : Retrieve historical execution traces, reconstruct the target policy’s familiar state distribution, and guide the robot into that distribution for stable handoff.

Evolution Skills : Update policy metadata, parameters, and harness logic online based on feedback.

Step 1: Identifying Dynamic Capability Boundaries

Capability boundaries are not static; they vary with camera pose, image quality, and robot state. For example, VLA may fail when image quality degrades. RoboHarness’s Understanding Skills compute multi‑dimensional metrics (semantic similarity, pose uncertainty, exposure) to predict which controller is most likely to succeed under the current conditions.

Step 2: Solving the Handoff Gap

Because each controller is designed independently, the final state of one may be unseen by the next, causing a “handoff gap.” The Memory Bridge retrieves successful trajectories for the upcoming policy from multimodal memory, extracts end‑effector pose and joint states, and fits a spatial distribution of "robot state — execution progress." It then selects a target state based on confidence and motion cost, generates a bridging trajectory, and transfers control.

Empirical Evaluation

On the LIBERO‑Plus benchmark with multiple disturbance types, RoboHarness adjusts policy usage according to observed reliability: it favors VLA when stable and routes to TAMP when perception or state falls outside VLA’s boundary. This dynamic orchestration yields a 93.2% average success rate, surpassing existing methods.

Long‑horizon evaluation on LIBERO‑LoHo (four times longer than standard LIBERO) shows that a single policy (π0.5) achieves only 6.4% success zero‑shot, and even hierarchical approaches (world model, LLM‑guided, logic‑guided) reach at most 64.8%. RoboHarness, by coordinating three policies (π0.5, RL‑trained OpenVLA, and TAMP), raises full‑task success to 95.2%.

Ablation Study

Removing Understanding Skills leads to incorrect task decomposition and routing because the system cannot assess policy boundaries. Eliminating the Memory Bridge causes failures even when the correct policy is chosen, due to incompatible handoff states. Omitting Evolution Skills prevents online refinement of capability estimates, degrading performance. The study confirms that understanding, adaptation, and handoff are all indispensable.

Implications

The authors view heterogeneous policy orchestration as a pragmatic path toward embodied intelligence: rather than chasing a monolithic universal model, systems should recognize each strategy’s reliability envelope, allocate tasks accordingly, and ensure seamless transitions. RoboHarness also accumulates cross‑policy, long‑horizon data that can later inform the training of more unified embodied models.