Skip to content
109M JEPA Beats 72B Qwen: LeCun Shows Pairing Strategy Trumps Scale for Alignment

109M JEPA Beats 72B Qwen: LeCun Shows Pairing Strategy Trumps Scale for Alignment

LeCun and Balestriero prove image and text models trained on different views of the same scene align via a single rotation; a 109M text JEPA outperforms 72B Qwen on image-text alignment by pairing captions of the same image instead of next-token prediction.

SmartApe AI
SmartApe AI
SmartApe AI

Paper Overview

On October 29, 2024, Randall Balestriero and Yann LeCun published the paper "Proving and Improving the Platonic Representation Hypothesis with JEPAs" on arXiv. The work proves that under specific conditions, image and text models trained separately on different observations of the same scene can align their representations via a single rotation, even without joint training.

Key Result: Parameter Efficiency

A text-only JEPA model with 109 million parameters outperformed the 72-billion-parameter Qwen2.5-72B on image-text representation alignment — a 660× parameter gap . LeCun noted that large models answer many questions because they memorize vast factual knowledge, whereas JEPA world models aim to learn regularities, reducing reliance on knowledge accumulation.

Theoretical Foundation: Platonic Representation Hypothesis

In 2024, Isola and colleagues proposed the Platonic Representation Hypothesis : as vision and language models scale, they converge to measuring distances between concepts in similar ways. LeCun and Balestriero asked whether training data pairing could accelerate this convergence. They define "good views" as pairs of samples (e.g., two photos of the same scene from different angles, or two captions describing the same image) that share the same underlying scene.

Experimental Comparison on Flickr

Researchers compared three text-only training strategies on Flickr:

Paired captions of the same image (different descriptions of the same visual scene)

Masked token prediction (masking different parts of the same sentence)

Next-token prediction (standard language modeling)

All three started near random alignment. As training progressed, the paired-caption strategy pulled ahead. With fixed encoder size, training text, and sample count:

Paired captions: alignment score 0.184

Next-token prediction: 0.111

Masked token prediction: 0.088

Sample Efficiency

The JEPA model reached 0.109 alignment after seeing 21.6 million samples , matching the next-token prediction model's 0.111 after 900 million samples — a 42× sample reduction (≈ 14× fewer tokens processed).

Scaling to Web Data

Training on captions from 50 million web images, a 33-million-parameter text JEPA achieved 0.125 on WIT, on par with Qwen2.5-3B (3 billion parameters) .

Challenge: Pure Text Corpora (Common Crawl)

Nicolas Dufour asked whether the advantage holds when training on plain web text (Common Crawl) without image-linked captions. In large-scale experiments, the text JEPA used three text segments per image: the original caption plus two LLaVA-generated descriptions. Results on 2.24 million images:

Original caption + LLaVA descriptions: 0.107

Original caption + pure-text rewrites: 0.060

Masked original caption: 0.055

Both the visual grounding in LLaVA's descriptions and the pairing strategy contributed. Balestriero acknowledged that pairing the same text differently still helps, but pure-text scenarios need better methods to construct "good views."

Demo: Zero-Shot Image Captioning

The paper demonstrates alignment by encoding an image, applying the learned rotation to map visual features into the text space, and decoding a description. Isola tested a photo of his cat Moxie and reported the output was "completely accurate."

The core insight: which samples are paired during training matters more than raw scale for cross-modal alignment. A tiny model trained on well-paired "good views" can align with a massive model trained on next-token prediction.
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Self-Supervised LearningRepresentation Learningmultimodal alignmentsample efficiencyimage-text alignmentYann LeCunJEPAPlatonic Representation Hypothesis
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.