VLDA: Learning Task-Relevant State Changes for Robot Manipulation

Baidu Intelligent Cloud and Harbin Institute of Technology (Shenzhen) introduce VLDA (Vision-Language-Delta-Action), a paradigm shift for robotic manipulation that replaces full-future prediction with learning a compact Delta variable representing only the task-relevant state change. The work is open-sourced in the LoongForge training framework.

1. The Core Problem: What Variable Should Robot Manipulation Learn?

Existing approaches fall into two camps:

VLA (Vision-Language-Action) maps observations directly to actions, forcing a single policy to simultaneously recognize objects, infer goal states, and generate action chunks. The semantic-to-physical gap is implicit, making long-horizon tasks hard to optimize.

WAM (World-Action Model) predicts complete future states (frames or latents) as an auxiliary objective. This wastes capacity on static backgrounds and task-irrelevant changes, and the predicted future remains a passive context rather than an actionable intermediate variable.

Both lack an explicit, structured representation of the task-relevant transition — the minimal physical change the robot must effect. VLDA posits that an ideal transition representation must be:

Selective — focus only on changing regions, ignore static background.

Predictive — align with the overall future trajectory, not just local details.

Actionable — directly condition action generation, not just serve as auxiliary supervision.

2. VLDA Architecture: Delta as the Transition Interface

The action generation pipeline is factorized into three stages:

Vision-Language Model (VLM) encodes current observation and language instruction into a multimodal context.

World Expert infers Delta — the required state change — from the context.

Action Expert generates an executable action chunk conditioned on Delta.

During training, Delta is supervised by future states; at inference, it is predicted solely from current observation and instruction. The V-JEPA 2 asymmetric encoder (online encoder for current frame, EMA teacher for future frame) stabilizes the supervision target.

3. Three Components Realizing the Delta Requirements

Local Transition Filter (LT-Filter) — Selectivity

Extracts dense semantic features from current and future frames, computes their difference, and selects the top- K regions with largest change (K=64). Unordered regions are matched via entropy-regularized optimal transport (Sinkhorn), yielding consistent supervision regardless of region ordering. Ablation: Sinkhorn matching achieves 98.5% vs. 97.2% (MSE) and 97.8% (Chamfer).

Global Transition Anchor (GT-Anchor) — Predictivity

Adds a coarse future evolution summary as a regularization constraint, ensuring Delta captures both local changes and global spatial trajectory. Complements LT-Filter's locality.

Delta-Driven Action Refinement (DDAR) — Actionability

Delta-to-Action Manifold Grounding (DAMG) : Contrastive learning aligns Delta with the current ground-truth action chunk (positive) against other timesteps' actions (negatives). Prevents visually plausible but control-irrelevant representations.

Delta-Seeded Action Flow (DSAF) : Injects aligned Delta as both a noise bias (rewriting flow-matching initial point) and a prefix token throughout the denoising process. Ablation: noise bias alone 97.9%, prefix token alone 97.6%, combined 98.5%.

4. Two-Stage Training on LoongForge

Stage 1 (Transition Pre-training) : Freeze action branch; optimize LT-Filter and GT-Anchor objectives to stabilize Delta representation.

Stage 2 (Joint Optimization) : Unfreeze all modules; jointly optimize transition modeling, action generation, and Delta-action alignment.

Model backbone: π0.5 (PaliGemma 2B VLM + Gemma-300M action expert) plus a newly added World Expert of identical architecture initialized from scratch. LoongForge supports DDP, ZeRO-1, FSDP, HSDP, component-wise parameter freezing, and checkpoint management.

5. Experimental Results

Simulation Benchmarks

LIBERO : 98.5% average success (Spatial 99.8%, Object 99.6%, Goal 99.4%, Long 95.2%), surpassing π0.5 by 1.0–2.8 pp. Largest gain on Long suite (+2.8 pp) indicates benefit for multi-step coherence.

LIBERO-Plus (robustness/generalization under 7 perturbation dimensions): 84.5% zero-shot success vs. π0.5 73.6% (+10.9 pp) and GuidedVLA 75.4% (+9.1 pp). Lighting perturbation: VLDA 96.0% (highest), confirming LT-Filter's immunity to appearance changes.

RoboTwin 2.0 (cross-distribution generalization): 62.9% on 10 hard tasks vs. π0.5 42.7% (+20.2 pp); 50.86% on all 50 tasks vs. π0.5 43.84% (+7.02 pp). Large gains on tasks requiring multi-step state changes (e.g., Put Bottles Dustbin +46 pp).

Real-Robot Evaluation

Two platforms: GALAXEA R1 Lite and AgileX Cobot Magic (ALOHA). Four long-horizon tasks (ring stacking, table clearing, conveyor sorting, basket transport), 25 trials each.

GALAXEA R1 Lite: VLDA 73.0% vs. π0.5 56.0% (+17.0 pp) vs. OpenVLA-OFT 16.0% (+57.0 pp).

AgileX Cobot Magic: VLDA 72.0% vs. π0.5 54.0% (+18.0 pp) vs. OpenVLA-OFT 17.0% (+55.0 pp).

Better long-horizon consistency: VLDA's late-stage success drops less than baselines.

Spatial generalization (basket transport with OOD mat positions): VLDA 72% final placement success vs. baseline 48% (+24 pp), validating GT-Anchor's global trajectory awareness.

6. Ablation Studies: Why Learning Change Beats Predicting Future

Replacing Delta with full-future prediction: LIBERO drops 0.7 pp (98.5% → 97.8%), LIBERO-Plus drops 3.4 pp (84.5% → 81.1%), with lighting -6.9 pp and background -4.7 pp. Confirms that focusing capacity on task-relevant change outperforms full-frame prediction, especially under distribution shift.

Removing DSAF: -1.3 pp on LIBERO (98.5% → 97.2%), showing Delta must be injected into action generation, not just used as auxiliary loss.

Removing DAMG: -0.3 pp on LIBERO but -2.5 pp on LIBERO-Plus (lighting -6.3 pp, background -3.7 pp), proving action-grounding prevents Delta from drifting with visual distractors.

LT-Filter or GT-Anchor alone help; combined best. Basket transport (layout variation) jumps from 48% to 72% with both.

Skipping Stage 1 pre-training: LIBERO 97.3% (-1.2 pp), LIBERO-Plus 79.4% (-5.1 pp), confirming staged training stabilizes transition representation.

7. Deployment Efficiency

At inference, only current observation and instruction are needed. The two V-JEPA 2 encoders and projection heads used only for training-time target construction and alignment are discarded. Deployed components: VLM backbone, World Expert, conditioning module, Action Expert.

8. Conclusion

VLDA demonstrates that a compact, explicit Delta variable — selective, predictive, and actionable — serves as a more effective transition interface than either direct observation-to-action mapping or full-future prediction. The approach simultaneously improves long-horizon consistency, robustness to distribution shift, and generalization to novel scenes, suggesting that the key representation gap in embodied AI lies between semantic understanding and physical execution.