VLDA: Learning Compact State Deltas Beats Full Future Prediction for Robot Manipulation

Harbin Institute of Technology (Shenzhen) and Baidu introduce VLDA (Vision-Language-Delta-Action), a new paradigm for robotic manipulation that shifts from predicting full future observations to learning only the task-relevant state changes, called Delta. The work is open-sourced in the LoongForge training framework.

Problem: The Representation Gap in Robot Manipulation

Existing approaches fall into two camps. Vision-Language-Action (VLA) models map observations directly to actions, forcing a single policy to simultaneously identify objects, infer goal states, and generate action chunks without an explicit interface for "what changes." World-Action Models (WAM) predict full future frames or latents, but waste capacity on static backgrounds and task-irrelevant variations, and the predicted future typically serves only as an auxiliary objective rather than a structured variable that directly drives action generation. Both lack a dedicated, actionable representation of the task-relevant physical transition.

Core Idea: Delta as a Transition Interface

VLDA introduces a low-dimensional Delta variable that acts as an explicit "transition interface" between vision-language understanding and action generation. The pipeline decomposes into three stages: (1) a Vision-Language Model (VLM) encodes current observation and language instruction into a multimodal context; (2) a World Expert infers Delta — the required state change; (3) an Action Expert generates an executable action sequence conditioned on Delta. This "understand, transition, execute" loop makes Delta the sole explicit physical-change variable: supervised by future states during training, but inferred from only current observation and instruction at inference.

Three Modules Realizing Selective, Predictive, Actionable Delta

1. Local Transition Filter (LT-Filter) — Selectivity

LT-Filter extracts dense semantic features from current and future frames, computes their difference, and selects the top-K regions with largest changes (K=64; increasing from 16 to 64 improves performance, 100 yields no further gain). Static background is discarded. Because selected regions form an unordered set, entropy-regularized optimal transport (Sinkhorn) establishes soft correspondences invariant to indexing order. Ablation: replacing optimal transport with pointwise MSE or Chamfer distance drops average success rate from 98.5% to 97.2% and 97.8% respectively.

2. Global Transition Anchor (GT-Anchor) — Predictivity

LT-Filter alone risks over-focusing on local changes and losing global transition direction. GT-Anchor adds a coarse future-evolution summary as a regularization constraint, ensuring Delta explains both local changes and overall future trend. The two components complement each other: local selectivity plus global consistency.

3. Delta-Driven Action Refinement (DDAR) — Actionability

DDAR injects Delta into action generation via two sub-modules: (a) Delta-to-Action Manifold Grounding (DAMG) uses contrastive learning to align Delta representations with current action chunks (positive) against other timesteps in the same trajectory (negative), ensuring Delta encodes control-relevant changes, not just visual ones. (b) Delta-Seeded Action Flow (DSAF) feeds aligned Delta as a structured seed into flow-matching-based action generation: as a noise bias rewriting the initial point, and as a prefix token throughout denoising. Ablation: noise bias alone 97.9%, prefix token alone 97.6%, combined 98.5%.

Two-Stage Training on LoongForge

Model backbone: π0.5 (PaliGemma 2B VLM + Gemma-300M action expert) plus a newly added World Expert of identical architecture initialized from scratch. Training on 8×NVIDIA A800-SXM4 (80 GB). Stage 1 (transition pre-training): freeze action branch, optimize LT-Filter and GT-Anchor objectives to stabilize Delta representation. Stage 2: joint optimization of transition modeling, action generation, and Delta-action alignment. This staging prevents action-branch gradients from disrupting the nascent transition representation. Supervision uses an asymmetric V-JEPA 2 design: online encoder for current observation, EMA teacher for future observation, stabilizing the target space. At deployment, the two V-JEPA 2 encoders and projection heads are discarded; only VLM backbone, World Expert, conditioning module, and Action Expert remain. Inference runs on a single 32 GB consumer GPU.

Experimental Results

LIBERO (standard manipulation benchmark)

VLDA achieves 98.5% average success rate (Spatial 99.8%, Object 99.6%, Goal 99.4%, Long 95.2%), all state-of-the-art. Gains over π0.5: +1.0, +1.4, +1.4, +2.8 pp respectively. Largest improvement on Long suite indicates Delta benefits multi-step coherent tasks.

LIBERO-Plus (robustness/generalization under 7 perturbation dimensions)

Zero-shot evaluation over 10,030 rollouts: VLDA 84.5% vs π0.5 73.6% (+10.9 pp) and GuidedVLA 75.4% (+9.1 pp). On Long suite: +13.4 pp and +12.1 pp. Lighting perturbation: VLDA 96.0% (highest across dimensions), confirming LT-Filter's immunity to appearance changes that don't affect task essence.

RoboTwin 2.0 (cross-distribution generalization)

Trained on clean setting, evaluated on randomized setting (zero-shot). On 10 hard tasks: VLDA 62.9% vs π0.5 42.7% (+20.2 pp), winning or tying all ten. On all 50 tasks: 50.86% vs 43.84% (+7.02 pp). Largest gains on tasks requiring multi-step state transitions (Put Bottles Dustbin +46 pp, Stacking Three +30 pp, Place Dual Shoes +27 pp, Open Microwave +27 pp).

Real-Robot Evaluation

Four long-horizon tasks (ring stacking, tabletop sweeping, conveyor sorting, basket transport), 25 trials per task per platform, progressive checkpoints. GALAXEA R1 Lite: VLDA 73.0% vs π0.5 56.0% (+17.0 pp) vs OpenVLA-OFT 16.0% (+57.0 pp). AgileX Cobot Magic (ALOHA): VLDA 72.0% vs π0.5 54.0% (+18.0 pp) vs OpenVLA-OFT 17.0% (+55.0 pp). VLDA degrades less in later stages, showing better long-horizon consistency. Basket transport with OOD target positions: VLDA 72% final-stage success vs baseline 48% (+24 pp), demonstrating GT-Anchor's value for spatial generalization.

Ablation Studies: Why Learning Only Changes Wins

Replacing Delta modeling with full-future prediction drops LIBERO from 98.5% to 97.8% (−0.7 pp) and LIBERO-Plus from 84.5% to 81.1% (−3.4 pp), with lighting −6.9 pp and background −4.7 pp. This confirms that concentrating capacity on task-relevant changes outperforms full future prediction, especially under distribution shift. Removing DSAF drops LIBERO to 97.2%, proving Delta must be injected into action generation, not just used as auxiliary supervision. Removing DAMG drops LIBERO-Plus 2.5 pp (lighting 96.0%→89.7%, background −3.7 pp): without action-level constraints, Delta drifts toward visual changes irrelevant to control. LT-Filter and GT-Anchor each help; combined they excel, especially on layout-varying tasks (real-robot basket transport 48%→72%). Skipping stage-1 pre-training (joint from start) drops LIBERO to 97.3% and LIBERO-Plus to 79.4%, confirming staged training is essential for stable transition representation.

Conclusion

VLDA changes the modeling target: instead of observation→action or full future prediction, it learns a compact, explicit Delta variable representing task-relevant state change and makes it the central interface between semantic understanding and physical execution. This single shift simultaneously improves long-horizon consistency, robustness to distribution shift, and generalization to novel scenes. The results suggest the core difficulty in robot manipulation may not be model scale but a missing representation: a dedicated, executable "change" variable bridging semantics and action. A focused intermediate representation beats a complete future prediction for control effectiveness.