Can VLA‑JEPA Achieve Robust Vision‑Language‑Action with Few Robot Trajectories and Lots of Human Video?

VLA‑JEPA proposes a JEPA‑style latent world‑modeling framework for Vision‑Language‑Action (VLA) systems, moving prediction from pixel space to latent representation space. It unifies human video and robot demonstration data under a single training objective: the VLA backbone extracts a latent action token from the current observation, and a predictor forecasts the future latent state, which is aligned with a target encoder’s representation of the future frame.

The authors identify four fundamental issues with existing pixel‑level latent‑action methods: (1) pixel‑wise targets bias representations toward appearance rather than action; (2) real‑world videos amplify irrelevant motion noise such as camera motion and background changes; (3) information leakage allows latent actions to encode future frames directly, degrading semantic meaning; and (4) multi‑stage pipelines increase engineering complexity and cause mismatched objectives across stages.

To address these problems, VLA‑JEPA adopts Qwen3‑VL as the VLM backbone and introduces a learnable latent action token that models state transitions. Video frames are encoded by a V‑JEPA2 encoder into world‑state embeddings; a predictor conditioned on the current state and latent action predicts the future latent state, which is then aligned with the target encoder’s future state. When robot action annotations are available, a flow‑matching action head generates continuous end‑effector trajectories, allowing a two‑step training process: JEPA pre‑training followed by action‑head fine‑tuning.

Experiments evaluate VLA‑JEPA on three simulation benchmarks (LIBERO, LIBERO‑Plus, SimplerEnv) and a real‑world Franka robot setup. Pre‑training uses ~220k human videos from Something‑Something‑v2 and ~76k high‑quality robot trajectories from DROID. Fine‑tuning on LIBERO/LIBERO‑Plus uses ~2k simulated expert demonstrations, while the real‑robot experiments involve 100 tabletop grasp‑and‑place demonstrations. VLA‑JEPA achieves 97.2% average success on LIBERO and 78.1% on LIBERO‑Plus, outperforming strong baselines such as OpenVLA‑OFT and pi0.5, especially under out‑of‑distribution perturbations. On SimplerEnv it reaches 65.2% and 57.3% success rates, and on the real Franka robot it uniquely learns a second‑attempt grasp after an initial failure, a behavior absent in competing models.

An ablation study varying the proportion of human video data shows that increasing human video scale consistently improves robustness on LIBERO‑Plus across multiple disturbance dimensions. The authors attribute this to human video providing a “world dynamics prior” that stabilizes the model rather than directly adding new action capabilities.

In conclusion, VLA‑JEPA demonstrates that embedding human video as a source of latent world dynamics, together with a compact latent‑action representation, yields a more data‑efficient and robust VLA system. The work highlights the complementary roles of robot data (providing executable action grounding) and large‑scale human video (supplying diverse dynamic priors), suggesting future research should focus on converting broad visual world experience into controllable, predictive latent models for embodied intelligence.