Why the “L” in VLA Isn’t Redundant: Position‑Aware Rebinding Boosts Instruction Generalization 20‑40%

Background and Problem

Vision‑Language‑Action (VLA) models are a leading approach to embodied intelligence, yet they suffer from poor instruction generalization. Even simple paraphrases cause dramatic performance drops, e.g., VLA‑Adapter loses 51% on the LIBERO‑Para benchmark.

Goal

The desired capability is a robot that executes the same task correctly despite variations in linguistic expression, while keeping the visual scene, objects, and success conditions unchanged.

Proposed Solution: Grounded Semantic Re‑Binding (GSR)

A joint team from Shanghai Jiao‑Tong University and WuJie Power Lab proposes a simple yet effective fix: redesign the entry of language semantics into the visual‑action computation pipeline so the model no longer depends on massive paraphrase data.

Key Steps

Freeze a T5 encoder to process language instructions alone, isolating task semantics from visual and state inputs.

Inject the projected T5 semantics back into the model at architecture‑specific fusion points, respecting each model’s original multimodal flow.

Re‑initialize and train the Action Expert to learn the new language conditioning from scratch.

Benchmark and Baselines

The authors construct LIBERO‑Para, a language‑generalization benchmark derived from LIBERO‑Goal, containing 4,092 episodes with 870 action‑expression rewrites, 259 object‑description rewrites, and 2,963 combined rewrites.

Evaluated models on original versus paraphrased instructions:

SmolVLA: 72.0% → 4.47% success.

VLA‑Adapter: 98.2% → 46.82% success.

π0.5 (large pretrained): still shows a noticeable drop.

Analysis 1 – Task Structure Persists

Action‑level retrieval experiments generate an action chunk from a paraphrased instruction and compare it to ten candidate actions from original instructions. Retrieval@1 for all three models is far above random (0.1) and Semantic Retention is positive, indicating that task‑relevant information remains in the failed actions.

Analysis 2 – Layer‑wise Feature Intervention

For VLA‑Adapter, the authors replace the language feature at the last Bridge‑Attention block with the feature from the original instruction while keeping image, state, and other tokens unchanged. This restores 96.8% of the action difference and raises closed‑loop success from 60% to 96%.

In contrast, a single‑layer replacement recovers only ~10.5% for SmolVLA and ~31.3% for π0.5, showing that VLA‑Adapter concentrates language control in a specific block, whereas the other architectures distribute it across many layers.

Analysis 3 – Visual‑Language Coupling

By fixing the visual input to a constant image while allowing the language to vary, success improves from 46.82% to 61.58%, demonstrating that dynamic visual input amplifies the harmful effect of wording changes.

Analysis 4 – Removing Word‑Direction

Cross‑validation identifies 32 discriminative directions that separate original from paraphrased instructions. Removing these directions from paraphrased features reduces action difference from 0.4361 to 0.2282 and lifts success on 20 closed‑loop episodes from 55% to 90%. Random directions have no effect.

Results of GSR

All GSR models are trained only on original LIBERO‑Goal instructions.

VLA‑Adapter full‑para success: 46.82% → 70.94% (+24.12 pts).

SmolVLA full‑para success: 4.47% → 49.12% (+44.65 pts).

π0.5 full‑para success: 73.60% → 75.59%; PRIDE score reaches 70.4, surpassing Xiaomi‑Robotics.

Ablation where a T5 is simply added without re‑binding yields negligible gains (VLA‑Adapter 46.82% → 47.31%; SmolVLA +13.49%), confirming that the reorganized information flow is essential.

Language‑Route Authority Experiments

Four conditions test which language path controls actions:

C/C (both correct): 47.31% success.

W/C (native wrong, T5 correct): success drops to 5.11%.

C/W (native correct, T5 wrong): success remains 44%.

Thus the native Qwen path still dominates control in VLA‑Adapter, while SmolVLA suffers severe interference from dual language routes. π0.5 relies mainly on its native PaliGemma path.

Exploration: ParaVLA

ParaVLA uses frozen T5 for language and DINOv2 for vision, keeping them largely independent until the Action Expert. It achieves 92% (original) and 91% (paraphrased) success, a 1‑point gap. Replacing T5 with a SmolVLM decoder drops paraphrased success to 41%, reinforcing the benefit of a stable, vision‑independent language source.

Discussion and Conclusion

GSR does not merely add more data or a language encoder; it reconstructs where language semantics enters the model and which pathway ultimately drives actions. The study shows that instruction‑generalization failures often stem from subtle semantic drift during multimodal fusion rather than a total lack of task understanding. Future work will aim to preserve stable semantics while retaining rich scene grounding and scaling to larger multimodal models.