Yann LeCun: Predicting Pixels is a Fallacy, JEPA Predicts in Abstract Space

We Are Far from AGI: LeCun's Opening Assessment

Yann LeCun opens his ECCV 2026 keynote by stating bluntly that current AI "sucks" compared to biological intelligence. While LLMs excel at coding, math, and exams, they fail at physical-world tasks: no general-purpose household robot exists, and L5 autonomous driving remains unsolved. The core issue is that AI systems learn with vastly lower efficiency than animals or humans. A 17‑year‑old learns to drive in ~20 hours with almost zero crashes; end‑to‑end imitation learning on billions of hours of driving data has not produced L5 autonomy. Near‑L4 systems rely on heavily engineered explicit world models, not pure data‑driven learning.

Moravec's Paradox and the Definition of Intelligence

LeCun invokes Moravec's Paradox (1988): computers solve complex integrals and chess but struggle with simple sensorimotor skills that animals master effortlessly. He adopts a behavioral definition of intelligence: "Intelligence is not what you know, but what you do when you face a novel situation." True intelligence is the ability to invent new skills for new problems — zero‑shot or few‑shot adaptation — not the accumulation of declarative knowledge and narrow skills. Humans are not "general" in the mathematical sense; each person masters only a tiny fraction of possible tasks. Therefore, AI should not aim for omniscience but for rapid learning and broad adaptability.

Text‑Only Training Is Information‑Theoretically Insufficient

LeCun quantifies the data gap. A typical LLM pre‑trains on ~30 trillion tokens (~10¹⁴ bytes). A four‑year‑old child receives roughly the same volume of visual data: 2 million optic‑nerve fibers × 1 byte/second × 60,000 waking hours ≈ 10¹⁴ bytes. Yet the child understands intuitive physics, causality, and 3D structure — capabilities LLMs lack. Conclusion: text alone cannot yield human‑level intelligence; visual and sensorimotor data are essential.

From Token Prediction to World Models: System 2 Is Missing

Current LLMs perform only feed‑forward next‑token prediction — a passive, System‑1‑like reaction with no internal reasoning. They compensate via Chain‑of‑Thought (generating many tokens to increase compute) and best‑of‑N sampling with verifiable rewards (code, math). But in the physical world, verification is slow and often incompressible; simulators work for locomotion but not for manipulation. LeCun proposes an energy‑based inference architecture: a network scores candidate answers (actions) with a scalar energy; external search/optimization finds the minimum‑energy action sequence. This iterative propose‑evaluate‑refine loop is classic probabilistic inference and corresponds to human System 2 thinking.

World Models and Hierarchical Planning

A world model predicts the next state given current state and a hypothetical action. Coupled with a cost function, it enables model‑predictive control (MPC): search for action sequences that minimize cost. LeCun argues the next frontier is hierarchical planning : humans plan at high abstraction levels ("go to airport") and recursively decompose into sub‑goals down to millisecond muscle commands. This requires hierarchical world models that operate at multiple timescales and abstraction levels — a key unsolved challenge.

JEPA: Joint Embedding Predictive Architecture

Generative video prediction (predicting pixels) is fundamentally impossible because the information needed for the next frame (e.g., a person walking into view) is not present in the current frame. Probability distributions over high‑dimensional pixel spaces are intractable; models collapse to blurry averages. JEPA avoids this by predicting in an abstract representation space . An encoder discards unpredictable details (leaf motion, water ripples, occluder appearance) and retains only predictable, plannable structure. The predictor operates on these latent representations, making the task feasible.

Preventing Representation Collapse

If trained only to minimize prediction error, the encoder can output constant vectors (collapse). LeCun reviews collapse‑prevention methods dating back to 1990s Siamese networks. Two families: sample‑contrastive (push sample representations apart, requires huge batches) and dimension‑contrastive (make feature dimensions statistically independent). He favors dimension‑contrastive methods: VICReg (variance‑invariance‑covariance regularization), SIGReg (enforces independent Gaussian marginals via linear projections and CDF matching), and DINO (clustering‑based). SIGReg projects representations to a higher‑dimensional space and forces each projected dimension to follow a standard Gaussian; this maximizes information while keeping only predictable structure.

Empirical Results: LeWorldModel, I‑JEPA, V‑JEPA

LeWorldModel : Uses SIGReg to learn world models for simple planning; can recover underlying Gaussian latent variables from complex sensor observations.

I‑JEPA : Self‑supervised image representation learning; outperforms supervised and CLIP features on dense tasks like depth estimation and segmentation.

V‑JEPA 2 / 2.1 : Video JEPA trained by masking large spatiotemporal tubes and reconstructing full video representations. Learns intuitive physics: prediction error spikes for impossible events (ball stopping mid‑air, turning into a cube), providing a "surprise" signal analogous to infant cognition studies.

Emergent 3D understanding : V‑JEPA 2.1 is monocular, trained only on video. A linear probe on its representations predicts depth maps better than methods using explicit depth supervision — the model infers 3D structure from motion parallax without ever being taught "depth."

Safety via Guardrails in Energy‑Based Architectures

Because the architecture optimizes actions to satisfy a cost function while respecting hard‑coded guardrail objectives (safety constraints), safety is built into the optimization process. Unlike LLMs where control is probabilistic and "jailbreaks" are possible, this system cannot take actions that violate guardrails — provided the world model and guardrails are correctly specified. LeCun acknowledges three failure modes: wrong world model, wrong guardrails, or wrong perception of world state.

LeCun's Provocative Recommendations

Stop chasing better generative models; switch to joint‑embedding architectures.

Advocate for joint‑embedding, abstract representations, and world models.

Prefer theoretically grounded regularization (dimension‑contrastive) over sample‑contrastive.

Use reinforcement learning judiciously — it is sample‑efficient for trial‑and‑error learning.

If you want human‑level AI, stop working on LLMs. They are useful for text/code/math but cannot bridge the physical‑world gap.

Q&A Highlights

LLMs still have value for text‑centric tasks, but they are insufficient for AGI.

Energy/water/hardware costs are political/societal issues, not technical ones; economists (including Nobel laureates) should guide policy.

Goal specification for robots can be images, text, or any encoding mapped to the same latent space.

Biological data modeling with JEPA is already being explored (cellular dynamics, MEG/EEG).

Irreversible actions : safety depends entirely on world‑model accuracy and guardrail design; no absolute guarantees exist.