Harness VLA Redefines Embodied AI Execution, Surpassing NVIDIA Cap‑X Performance

Problem

Vision‑Language‑Action (VLA) models achieve near‑saturation on standard benchmarks (e.g., Pi 0.5 reaches 96% success on 130 LIBERO tasks). However, when deployed in environments with target rebinding or layout disturbances (LIBERO‑Pro), success drops sharply from 96% to 50%, revealing a core generalization challenge.

Solution Insight

Recent coding agents (Claude Code, Codex) show that performance gains stem not only from larger models but also from an added Harness Layer that maintains context, invokes tools, handles failures, and decides when the model should act. Harness VLA adopts this paradigm for robotics.

System Architecture

The frozen VLA model is wrapped as a unified VLA_ACT primitive responsible for contact‑intensive actions (grasp, place, button press, drawer/door operation). Analytic primitives such as MOVE_TO, ROTATE_WRIST, SET_GRIPPER, and RELEASE handle stable spatial moves, pose adjustments, transport, and release. An Agentic Planner learns the boundaries of each primitive, determines necessary viewpoint and pre‑contact adjustments, and devises recovery strategies after failures.

Training and Deployment Phases

Exploratory Bootstrapping Phase : The system performs trial‑and‑error in the environment, records successful primitive sequences as Task‑Specific Memory, and aggregates cross‑task failure rules into Global Memory.

Deployment Evaluation Phase : The planner uses these memories to rebind targets, adjust states, and decide when and how to invoke primitives, without adding new primitives at deployment time.

Evaluation

Few‑shot on LIBERO‑Pro (Task‑Specific Memory from Seed 0 applied to other seeds) yields 82.4% success, compared to 50.0% for the baseline RLinf model, and far exceeds Pi_RLinf (50%), NVIDIA Cap‑X (18.2%), and Berkeley RATS (43.8%).

Zero‑shot on LIBERO‑Pro Goal shows 31.0% (position‑swap) and 79.0% (instruction‑redirection) success, outperforming Cap‑X (25.6% and 16.8%).

On RoboCasa365 (real kitchen tasks) Harness VLA lifts success from 60 % / 21.3 % / 5 % to 91.6 % / 56.3 % / 15 % across ATOMIC‑SEEN, COMPOSITE‑SEEN, and COMPOSITE‑UNSEEN.

On RoboTwin C2R (clean → randomized transfer) zero‑shot success reaches 58.4%, beating representative VLA (47.9%), GR00T‑N1.7 (20.7%), and StarVLA (10.6%).

Analysis of Failure Modes

Many deployment failures arise from acting on the wrong target, wrong state, or wrong task stage. Harness VLA’s planner rebinds the correct target, ensures the VLA is called in a suitable state, and uses feedback to decide whether a second call is needed. Analytic primitives handle non‑contact phases, allowing the frozen VLA to focus on contact‑dense skills.

Attribution studies show that in LIBERO‑Pro most successful rollouts conclude after an Analytic Primitive, whereas in RoboCasa365 and RoboTwin C2R they often finish after a VLA_ACT primitive, confirming the intended division of labor: Analytic primitives orchestrate overall task flow, while VLA_ACT provides essential contact actions.

Memory‑Driven Planning

Task‑Specific Memory stores successful primitive trajectories for each task; Global Memory captures reusable failure recovery rules. During deployment, the Agentic Planner queries these memories to decide when to invoke VLA_ACT, when to precede it with Analytic primitives (e.g., MOVE_TO for staging), and whether to retry after a failed contact attempt.

Effect of Primitive Call Limits

Experiments varying the maximum allowed VLA_ACT calls per episode show that success rates on LIBERO‑Pro, RoboCasa365, and RoboTwin C2R rise sharply within the first few calls and plateau near the full Harness VLA performance, indicating that only a few targeted calls are needed.

Open‑Source Release

The Harness VLA code and paper are publicly available:

Paper: https://arxiv.org/pdf/2607.08448

Repository (part of the RPent ecosystem): https://github.com/RLinf/RPent

RPent provides an agentic infrastructure for continual evolution of embodied AI, enabling modular composition of models, primitives, and memory components.