How VISReg Overcomes JEPA’s Representation Collapse – LeCun’s Endorsement

Background

Self‑supervised learning (SSL) learns general visual representations without manual labels, but a core difficulty is representation collapse : the model maps different inputs to the same or a few vectors, appearing trained while lacking discriminative features.

Existing Remedies

Most SSL methods mitigate collapse with heuristic tricks such as EMA, teacher‑student networks, stop‑gradient, or layer freezing, which make training fragile and hard to tune. VICReg replaces heuristics with a variance‑invariance‑covariance loss, but its covariance term only captures second‑order statistics and cannot distinguish distributions with identical mean and variance but different shapes. SIGReg adds a Cramér–Wold‑based sketching regularizer that aligns the whole embedding distribution to a standard Gaussian, yet it suffers from two critical flaws:

Gradient vanishing when collapse occurs, leaving the model unable to recover.

Coupling of scale and shape, causing interference on long‑tail, low‑quality, or low‑rank data.

VISReg Design

VISReg (Variance‑Invariance‑Sketching Regularization) combines the strengths of VICReg and SIGReg while addressing their weaknesses. It decouples the regularization into three independent components:

Scale Regularization

Constrains the variance of each dimension to avoid amplitude collapse. When the model collapses, the gradient of this term remains a constant, guaranteeing a stable recovery signal.

Shape Regularization

First normalizes the centered embeddings (stop‑gradient on the standard deviation) to remove scale influence, then applies a sliced Wasserstein distance (SWD) between the normalized embeddings and a standard Gaussian along many random projection directions. By the Cramér–Wold theorem, aligning all one‑dimensional projections forces the full high‑dimensional distribution to match, thus capturing the complete shape.

Combined Objective

Adds a centering loss that pulls the batch mean toward the origin. The three losses are summed with a single hyper‑parameter λ that balances prediction and regularization.

The full objective can be written as:

def visreg(z, K=64):
    # 1. Centering loss
    mu = z.mean(dim=0)
    L_center = (mu ** 2).mean()
    # 2. Scale loss
    z_cent = z - mu
    std = z_cent.std(dim=0, unbiased=False)
    L_scale = (1.0 - std).pow(2).mean()
    # 3. Shape loss: sliced Wasserstein distance
    z_norm = z_cent / (std.detach())
    W = torch.randn(D, K)
    W /= W.norm(p=2, dim=0)
    p_sorted = torch.sort(z_norm @ W, dim=0).values
    u = torch.arange(1, N+1) / (N+1)
    target = Normal(0, 1).icdf(u)
    L_shape = (p_sorted - target).pow(2).mean()
    return L_scale + L_shape + L_center

Complexity and Scalability

The regularization cost of VISReg is linear in batch size N, dimension D, and number of slices K (O(N·D·K)). By contrast, VICReg’s covariance term grows quadratically with D. VISReg therefore scales better to high‑dimensional embeddings. Moreover, the K random slices can be distributed across multiple GPUs: with M GPUs each generates K/M slices, achieving the same effect as a single‑GPU implementation while keeping per‑GPU workload constant.

In experiments, using 8 GPUs with 128 slices each (total 1024) reduced the accuracy gap caused by insufficient slices from ~2.4 % to 0.22 %.

Experimental Evaluation

VISReg was benchmarked on 15 datasets (8 in‑domain, 6 out‑of‑distribution, and ADE20K dense prediction) against seven mainstream SSL methods (MoCoV3, DINO, iBOT, I‑JEPA, MAE, data2vec, etc.).

In‑Domain Linear Probing

Without any heuristic tricks, VISReg achieved 75.7 % top‑1 accuracy with ViT‑B/16, surpassing MAE (75.1 %). With ViT‑L/14 the accuracy rose to 77.0 %, beating LeJEPA (75.6 %). On the texture dataset DTD, VISReg outperformed all competitors.

Out‑of‑Distribution (OOD) Generalization

On six OOD datasets (ChestXRay, RetinaMNIST, OrganAMNIST, Galaxy10, AID, DTD), VISReg attained the highest average OOD accuracy across all model sizes: ViT‑B/16 reached 70.19 % and ViT‑L/14 reached 70.63 %, exceeding DINO, MoCoV3, and others.

Data Efficiency

Pre‑training VISReg (ViT‑L/14) on ImageNet‑22K (~14 M images) achieved 72.94 % average OOD accuracy, matching DINOv2 trained on 10× more data (LVD‑142M). Thus VISReg reaches comparable performance with roughly one‑tenth of the data.

Fine‑Tuning Transfer

After fine‑tuning, VISReg outperformed DINO and supervised pre‑training on CIFAR‑10, CIFAR‑100, Flowers, ImageNet‑1K, and Galaxy10, indicating more uniform and less redundant representations.

Dense Prediction and Generation Guidance

On ADE20K linear semantic segmentation (ViT‑B/16), VISReg attained mIoU 30.16, surpassing DINO (29.40) and MAE (23.60). For generation guidance using SiT‑B/2 within the iREPA framework, VISReg‑guided images achieved lower gFID (40.36 vs 41.15), higher Precision (51.38 vs 50.51) and Recall (61.26 vs 60.70) than DINO.

Robustness on Low‑Quality Data

On long‑tail ImageNet‑LT and low‑rank Galaxy10, VISReg consistently prevented collapse and learned meaningful features, whereas DINO often failed without extensive hyper‑parameter tuning.

Conclusion

By decoupling representation regularization into independent scale and shape components via sliced Wasserstein sketching, VISReg provides a more stable, efficient, and generalizable SSL method. It achieves leading or competitive results on image classification, segmentation, and generation tasks without any heuristic tricks and matches DINOv2’s OOD performance using only one‑tenth of the training data, offering a promising solution to the long‑standing representation collapse problem in JEPA world models.