Beyond Binary Success: Redefining Fine-Grained Manipulation Evaluation for Embodied AI

Fine‑grained manipulation is a key capability for embodied intelligence, yet most existing evaluations rely on a binary "success/failure" metric that hides shortcomings in semantic understanding, precise perception, and stable execution.

The authors from Southeast University and Peking University propose MetaFine , a diagnostic meta‑evaluation platform that decomposes manipulation ability into three dimensions:

Understanding : Does the agent truly grasp the task semantics? For example, when the instruction changes from "grab the bottle cap" to "grab the bottle body", a model that understands attribute‑level language should correctly shift its target.

Perception : Can the agent perceive local structure under visual perturbations such as viewpoint or lighting changes?

Behavior : Does the agent execute a constrained, stable motion trajectory, e.g., inserting a letter block without collision or mis‑alignment?

To illustrate the challenge, the paper describes a simple letter‑block assembly task where a robot must (1) identify the missing letter, (2) locate the correct slot, and (3) insert it without collision. Failure in any sub‑step leads to overall task failure, highlighting the fragility of fine‑grained skills.

Finding 1 : Traditional coarse benchmarks can over‑estimate fine‑grained capability by up to 70 %. Experiments show that models achieving 80 % success on standard grasping tasks often fail when evaluated with MetaFine’s stricter part‑level and constraint‑level criteria.

Finding 2 : Failures are layered. The authors evaluated several Vision‑Language‑Action (VLA) models and observed distinct failure modes:

Understanding layer : Models often follow scene‑action correlations rather than truly interpreting the instruction, as shown when changing the target part does not alter behavior.

Perception layer : Loss of local spatial information in visual encoders directly limits manipulation performance; improving encoder fidelity unlocks previously impossible actions.

Behavior layer : Deterministic motion generators yield stable trajectories but can be brittle, while stochastic generators are expressive but may drift under perception uncertainty, illustrating a trade‑off between stability and flexibility.

MetaFine is not a single benchmark but a meta‑evaluation base. It builds composable task graphs where nodes represent atomic fine‑grained skills (e.g., grasp part, align, insert, press, rotate) and edges encode dependencies, allowing integration of existing benchmarks into the three‑dimensional diagnostic space.

To address the high cost and variability of real‑robot experiments, MetaFine supports hybrid evaluation: large‑scale simulation tests are calibrated with a limited set of paired real‑robot trials, enabling scalable yet physically credible performance estimates and cross‑lab comparability.

Overall, MetaFine shifts evaluation from ranking by outcome to diagnosing capability, providing researchers, benchmark designers, and robot deployers with actionable insights into whether failures stem from language understanding, visual perception, or motion control, thereby fostering the development of truly dexterous embodied agents.