World Models Ending Pixel Reconstruction: 14‑Paper JEPA Roadmap

Yann LeCun has pursued a research direction that diverges from the dominant trend of scaling language‑model parameters, focusing instead on building goal‑driven AI systems whose core is a world model that simulates environment dynamics without pixel‑level reconstruction.

Core Mechanism Overview

Current large models excel at capturing textual patterns but lack physical commonsense and multi‑step planning abilities. JEPA (Joint Embedding Predictive Architecture) addresses this by discarding pixel reconstruction and predicting future states directly in an abstract latent space. The encoder first transforms paired inputs (e.g., consecutive video frames) into abstract representations, discarding background noise and irrelevant details, and the predictor then forecasts the latent representation of the future target.

Stage 1 – From Theory to Image Validation

Before JEPA, self‑supervised visual learning relied on pixel reconstruction (e.g., MAE) or contrastive learning with heavy data augmentation. LeCun introduced the principle that prediction must occur in abstract representation space.

JEPA and H‑JEPA – early concepts that added hierarchical and multi‑timescale mechanisms to enable longer‑range state prediction.

I‑JEPA (Image‑based JEPA) – the first engineering realization. It samples four target patches (15‑20% area) and a larger context patch (85‑100% area), removes overlapping regions, and updates the target encoder via exponential moving average (EMA) of the context encoder. A lightweight predictor minimizes L2 distance between predicted and true target embeddings, achieving 91% linear‑probe accuracy on CIFAR‑10.

Stage 2 – Dynamic and Multimodal Extension

After validating on static images, the architecture was extended to handle temporal dynamics and cross‑modal data.

MC‑JEPA (Motion‑Content JEPA) – combines motion and content features in a shared encoder, demonstrating that abstract‑space prediction can capture both static details and dynamic changes.

V‑JEPA (Video‑based JEPA) – processes video as a sequence of 16×16 spatial tokens over 2 frames. It replaces L2 loss with L1 loss for stability, applies a 3‑D multi‑block mask that discards up to 90% of spatio‑temporal patches, and trains on the VideoMix2M dataset. V‑JEPA outperforms prior video‑mask models on Something‑Something‑v2 and runs roughly twice as fast as pixel‑reconstruction video models.

Audio‑JEPA – transfers the same prediction mechanism to audio spectrograms using time‑frequency masks, confirming the modality‑agnostic nature of the approach.

Stage 3 – 3D Geometry

Point clouds are unordered, making pixel‑level reconstruction inefficient. Point‑JEPA adapts JEPA to point‑cloud data by predicting latent features instead of raw coordinates, achieving efficient geometric representation.

3D‑JEPA – extends the framework to full 3D semantic learning, broadening applicability to complex spatial reasoning tasks.

Stage 4 – Action and Planning

To move from passive perception to active control, the system must model how actions affect the environment.

ACT‑JEPA – jointly predicts future observations and action sequences, improving task success rates in control benchmarks.

V‑JEPA 2 (Zero‑Shot Planning) – integrates action variables to enable zero‑shot robot planning: without fine‑tuning, the model can execute multi‑step visual sub‑goals in an unseen physical environment.

Stage 5 – Mathematical Refinement and End‑to‑End Modeling

Later work streamlines earlier engineering tricks (EMA, stop‑gradient) by grounding the objective in mathematics.

LeJEPA – replaces teacher‑student setups with isotropic Gaussian regularization (SIGReg), simplifying training and improving parallelism.

Causal‑JEPA – upgrades masking from patches to object‑level, forcing the model to infer masked objects from surrounding context, which boosts counterfactual reasoning and data efficiency.

V‑JEPA 2.1 – adds dense prediction loss (both visible and masked tokens) and deep self‑supervision across encoder layers, sharpening spatio‑temporal localization and enabling new performance baselines on short‑term prediction and fine‑grained scene understanding.

Stage 6 – Fully End‑to‑End World Model

LeWorldModel eliminates all auxiliary losses and external encoders, training directly from raw pixels with two objectives: next‑step feature prediction and Gaussian regularization. This minimal design reduces engineering complexity while delivering fast inference and strong physical‑consistency detection.

Stage 7 – Semantic Reasoning

ThinkJEPA incorporates semantic knowledge from vision‑language models into the latent prediction path, enabling long‑horizon planning and logical reasoning beyond low‑level feature extrapolation.

Across all stages, the JEPA family consistently demonstrates that abandoning pixel reconstruction in favor of abstract latent prediction yields higher training efficiency (e.g., V‑JEPA trains a ViT‑Huge/14 on 16 A100 GPUs in <1200 GPU‑hours), better label efficiency (stable fine‑tuning with only 5‑10 % labeled data), and superior performance on downstream tasks such as video action recognition, 3D semantic segmentation, and zero‑shot robot planning.

References: [1] BZ5a1r‑kVsf (OpenReview); [2] arXiv 2301.08243; [3] arXiv 2307.12698; [4] arXiv 2404.08471; [5] arXiv 2507.02915; [6] arXiv 2404.16432; [7] arXiv 2409.15803; [8] arXiv 2501.14622; [9] arXiv 2506.09985; [10] arXiv 2511.08544; [11] arXiv 2602.11389; [12] arXiv 2603.14482; [13] arXiv 2603.19312; [14] arXiv 2603.2228.