How Repositioning the Language Path Boosts VLA Instruction Generalization by 20‑40%

Vision‑Language‑Action (VLA) models achieve high success rates on benchmarks with fixed instruction wording, but their performance drops dramatically when the same task is expressed with paraphrased commands. For example, VLA‑Adapter’s success on LIBERO‑para falls by 51% after simple rewrites, and SmolVLA’s success drops from 72.0% to 4.47%.

Analysis 1: Residual Task Structure in Failed Actions

The authors design an action‑level retrieval experiment. For a given visual observation, the model receives a paraphrased instruction, generates an action chunk, and then, with the same observation, processes ten original LIBERO‑Goal instructions to obtain candidate actions. Retrieval@1 scores for all three models are far above random (0.1), indicating that even failed episodes retain task‑related information. This suggests that the loss of performance is not due to complete loss of instruction understanding but to the transformation of task information into actions.

Analysis 2: Layer‑wise Feature Intervention

Using VLA‑Adapter as a case study, the authors replace the language features at the last Bridge‑Attention block with those from the original instruction while keeping image tokens, state tokens, and all other inputs identical. This restores 96.8% of the action difference and raises paired‑episode success from 60% to 96%.

In contrast, performing the same single‑layer replacement on SmolVLA and π0.5 recovers only ~10.5% and ~31.3% of the action difference, respectively. The authors conclude that VLA‑Adapter concentrates language control in a dedicated Bridge‑Attention policy, whereas SmolVLA and π0.5 fuse language, vision, and state across many layers, dispersing task semantics.

Analysis 3: Visual‑Dynamic vs. Linguistic Perturbations

The authors isolate the effect of dynamic visual input on language representations by running two forward passes: one with the true image and another with a fixed dummy image while keeping the instruction unchanged. The language tokens from the dummy‑image pass are fed back into the original computation path. This manipulation raises VLA‑Adapter’s full LIBERO‑Para success from 46.82% to 61.58%.

Analysis 4: Removing Specific Linguistic Directions

Cross‑validation identifies 32 discriminative feature directions between original and paraphrased instructions. Removing these directions from unseen paraphrased commands reduces the action distance from 0.4361 to 0.2282 and lifts success on 20 closed‑loop episodes from 55% to 90%, while random direction removal leaves performance unchanged.

Proposed Solution: Grounded Semantic Re‑Binding (GSR)

GSR consists of three steps:

Freeze a T5 encoder to process instructions independently of images and robot state, extracting pure task semantics.

Inject the projected T5 semantics back into the model at architecture‑specific fusion points, replacing the original language pathway.

Re‑initialize and retrain the Action Expert so it learns to condition on the new, stable language signal.

Experiments show that GSR dramatically improves instruction generalization without using any paraphrased data during training. On LIBERO‑Para (4,092 episodes), GSR raises VLA‑Adapter success from 46.82% to 70.94% (+24.12 pts) and SmolVLA from 4.47% to 49.12% (+44.65 pts). π0.5 also gains a modest increase to 75.59% and achieves the highest PRIDE score (70.4), surpassing recent large‑scale pre‑trained models.

Adding a T5 encoder alone (Native + T5) yields negligible gains, confirming that the improvement stems from reorganizing the information flow rather than merely increasing model capacity.

Language‑Route Authority Experiments

Four conditions test which language path controls actions: C/C (both correct), W/C (native wrong, T5 correct), C/W (native correct, T5 wrong), and W/W (both wrong). VLA‑Adapter’s success drops to 5.11% when the native Qwen path receives a wrong instruction, but remains 44.0% when only T5 is wrong, indicating that the native path still dominates control. SmolVLA suffers severe drops in both conflict scenarios, showing stronger interference between the two language routes.

Exploring ParaVLA

ParaVLA freezes T5 for language and DINOv2 for vision, keeping them largely independent until the Action Expert. It achieves 92% success on original instructions and 91% on paraphrased ones, a 1‑point gap, whereas replacing T5 with a SmolVLM decoder drops paraphrased success to 41%.

Conclusion

GSR demonstrates that improving VLA instruction generalization is not solely a matter of data scaling; redesigning where and how language semantics enter the multimodal pipeline is crucial. Stable, vision‑independent language conditioning preserves task information, mitigates semantic drift, and yields consistent robot actions across varied linguistic expressions.