LIBERO-MAX: 224K Simulations Expose Robot Policy Fragility Under Mid-Task Changes

When a robot reaches for a cup and the cup is moved mid-motion, the robot must detect the change and adjust its ongoing plan. Existing robot benchmarks only test static variations — altered initial object positions, lighting, or camera angles — but do not capture the challenge of adapting after execution has begun.

Paired Benchmark Design

LIBERO-MAX constructs 8,000 paired test cases from LIBERO-Plus and LIBERO-PRO tasks. Each pair shares the same initial state, instruction, random seed, and pre-change actions. At a critical moment, the Base case continues without perturbation, while the Dynamic case introduces one of eight mid-execution disturbances: target object movement, container movement, camera viewpoint change, sensor noise, lighting switch, visual theme change, distractor addition, or obstacle insertion. This pairing isolates the effect of the disturbance from the policy's baseline competence.

Large-Scale Evaluation of 14 Policies

The benchmark evaluates 14 state-of-the-art policies spanning Vision-Language-Action (VLA) models, hybrid architectures, and World Action Models (WAM). Each policy runs 16,000 executions (8,000 Base + 8,000 Dynamic), totaling 224,000 simulations . All policies exhibit significant success-rate drops:

Success-rate decline ranges from 11.0 to 25.7 percentage points .

The best dynamic performers, MolmoAct2 and π₀.₅, achieve only 66.9% and 65.7% success respectively.

For one policy, over half of its originally successful Base cases become failures under Dynamic conditions.

Rankings interleave across model families; no single family dominates. World Action Models, despite their predictive capabilities, show similar vulnerability.

Disturbance-Type Breakdown

Decomposing results by disturbance type reveals that object/container movement and camera/sensor changes cause the largest drops , while lighting and visual theme changes have milder impact. This aligns with the intuition that geometric and observational shifts directly invalidate the policy's current action plan.

Action Chunking and Observation Frequency

Policies often predict action chunks executed over multiple steps before re-observing. The authors test 4 policies across 18 action-chunk intervals on 800 paired cases. While interval length affects absolute performance, Dynamic success rates remain below the corresponding Base rates in every setting — more frequent feedback does not automatically close the gap. A camera ablation further shows that the same viewpoint change is less harmful when present from the start than when introduced mid-execution, indicating that execution history influences recovery.

Targeted Mitigation: Image Quality Detection

On 300 fixed sensor-noise cases, an image-quality detection module is added before X-VLA: if quality is low, the image is denoised before policy inference. The policy is not retrained, and the detector receives no explicit change signal. Triggered repair raises Dynamic success from 40.7% to 47.7% (+7.0 pp) ; always-on repair yields +6.7 pp. Base performance is largely preserved. This demonstrates that LIBERO-MAX can quantify targeted fixes while checking for regressions on unperturbed tasks.

Lite Version and Open Resources

LIBERO-MAX releases a Lite subset of 800 pairs (10% of full executions) for rapid iteration, along with the full benchmark code, fixed case list, and paired diagnostics. Resources: paper (arXiv:2609.36518), code (github.com/liberomax/LIBERO-MAX), project page (liberomax.github.io).

The benchmark aims to make "world changed, can the robot still finish?" a core, reproducible question for robot learning research.