HumanCLAW Benchmark Shows VLMs Achieve Only 16.8% Success in Embodied Action Tasks

Foundation models are moving from merely understanding the physical world to participating in robot control stacks, taking on high-level decision roles while pre-trained controllers handle balance, trajectory tracking, and joint execution. This shift raises a key question: when low-level controllers reliably execute actions, does a vision-language model (VLM) truly know what the body should do next? It must not only see the target but continuously decide when to explore, how to approach, where to stop, and when to complete an interaction.

HumanCLAW Framework

Researchers from Meta, Nanyang Technological University, University of Washington, Brown University, and Northwestern University introduced HumanCLAW, a benchmark that isolates action intelligence — the ability to decide "what to do" rather than "how to move each joint." The framework preserves continuous motion, collisions, contact, and gravity while stripping away low-level control errors. It operates on a 0.5-second observe-decide-act-feedback loop with three components:

VLM Decision: A frozen off-the-shelf VLM selects one parameterized atomic skill per step (e.g., move forward, turn, sit) from a first-person view. No high-level shortcuts like "sit on the sofa"; the model must decompose finding, approaching, and orienting itself.

Continuous Motion Generation: A motion generator converts atomic skills into continuous whole-body motions. New skills are added via lightweight ControlNet adapters per skill, without retraining the motion model or VLM.

Semi-Physical Execution: The body executes actions reliably and never falls, but walls block paths, feet collide with furniture, and objects are knocked over. The model must confront the new state it just created. A single-step validator rejects obviously invalid actions before execution.

Benchmark Design

HumanCLAW-Bench comprises 1,218 long-horizon tasks across 41 indoor scenes. Each task requires three sequential phases: find the target, navigate within 20 cm of it, and complete a physical interaction (e.g., sitting). The benchmark records process metrics such as collisions and disturbances, not just success/failure.

Main Results

Nine state-of-the-art VLMs were tested under identical conditions with frozen weights and no task-specific training. The best model achieved phase success rates of 64.9% → 42.5% → 16.8% (Find → Navigate → Interact). Four of the nine models scored ≤0.2% on full interaction success. Performance drops sharply when a high-level plan must be unfolded into dozens or hundreds of continuous decisions: models often recognize the target and propose reasonable plans but fall behind during step-by-step execution.

Failure Mode Analysis

After target enters view, recognition is usually not the problem. Search failures stem from inefficient exploration — the model simply fails to bring the target into view.

After discovery, the body often cannot reach the target. 68% of navigation failures relate to first-person self-spatial reasoning: the model does not know how far it is from the target or when to stop.

Even after contact, interaction may fail. Sit-down success rates across models range from 90% down to 3.5%. The missing piece is not the sit action itself but issuing it at the correct moment.

Collision distribution reinforces this: legs collide in 28%–45% of action steps, while the head (near the visual center) collides in <7%. Models walk into walls they just correctly described, indicating a lack of embodied self-awareness — the inability to maintain an online estimate of "where is my body, which way is it facing, what did I just bump into?"

Ablation: Structured Reasoning Beats Longer History

Ablation experiments show that closed-loop action benefits from structured reasoning components, not longer context:

Compact structured memory, mid-level goals, and a short-context verifier all improve results. Removing mid-level goals drops full interaction success from 18.9% to 0%; removing the verifier cuts navigation success from 27% to 2%.

Adding more text or image history yields diminishing returns; ten frames of image history even degrades performance. The key is organizing information around body state: what just happened, how the world changed, what needs verification now.

Future Directions

Across diverse bodies and tasks: Extend morphologies, skills, object interactions, and task lengths to test generalization to open environments.

From simulation to real robots: Connect high-level decisions to learned controllers, reintroducing hardware noise, control errors, and real contact to measure how much decision capability is retained.

Continuous self-body estimation: Incorporate proprioceptive and contact signals, and train specifically for body-state estimation to determine whether embodied self-awareness arises from observation, memory, or dedicated learning mechanisms.

HumanCLAW provides a reusable method to track whether action intelligence scales with foundation model progress. If general reasoning transfers to action, the benchmark will record that change; if not, it will help pinpoint what remains missing. The question "Can foundation models become the decision layer of embodied intelligence?" shifts from vision to a testable research problem.

Paper: arXiv:2607.27180 (https://arxiv.org/abs/2607.27180) | Code: GitHub (https://github.com/Human-CLAW/HumanCLAW)