SADL Loss Eliminates JEPA Collapse Across All Seeds, Boosts Effective Rank 6x
Zhejiang University's SADL (Supply and Demand Loss) adds a single regularizer to JEPA training that reduces collapse rate from 50% to 0% across ten random seeds, increases encoder effective rank 3.4–6.4× after one epoch, cuts latent prediction error to 1/20–1/303, and accelerates convergence on world model and image classification tasks, earning a retweet from Yann LeCun.
JEPA Collapse Problem
JEPA (Joint Embedding Predictive Architecture) is a self-supervised learning approach championed by Yann LeCun. An encoder produces representations H, a projector maps H to endpoints Z, and a predictor forecasts the next Z from the current Z and action. Training aligns predictions with mean squared error, avoiding pixel reconstruction. However, JEPA suffers from representation collapse: the encoder maps diverse inputs to nearly identical vectors, making prediction trivial but representations useless. In LeWM and INTACT (two JEPA-style world models), roughly 5 out of 10 random seeds collapse.
SADL: Supply and Demand Loss
Researchers from Zhejiang University (Junhan Sun et al.) introduced SADL, a loss term added to the existing JEPA objective without changing architecture. The total loss combines the task loss, a demand regularizer D(Z) on endpoints, and a new supply regularizer S on encoder outputs H. No new networks or parameters are introduced.
Root Cause of Collapse
From the first training step, H is dominated by a component shared across all inputs; inter-sample differences account for only ~0.01% of H's energy. After one epoch with only the demand regularizer, this proportion remains 3–5%, while Z already shows 30–42% sample diversity. The cause lies in BatchNorm inside the projector: it subtracts the batch mean, placing the shared direction in the projector's Jacobian nullspace. Translating all H along this direction leaves Z unchanged. Since both the demand regularizer and task loss compute gradients from Z, they receive near-zero gradient energy (~10⁻¹⁰ and 10⁻⁹) along the shared direction, leaving it unchecked and gradually squeezing out sample differences.
SADL Mechanism
SADL shifts focus from "which regularizer is better" to "whether the encoder can supply enough diversity." It measures a demand capacity q at Z (geometric mean of demand regularizer satisfaction and current sample diversity at Z) and a supply capacity k at H (minimum of encoder's ability to excite sample differences and those differences' ability to propagate to Z). Only when k < q does SADL penalize the encoder; otherwise the term is zero, leaving a healthy encoder untouched. This one-sided gate makes SADL stable across two orders of magnitude of its own weight without per-setting retuning.
World Model Experiments
On four tasks (LeWM and INTACT with various demand regularizers), training for one epoch shows dramatic separation. Baselines with only demand regularizers fluctuate around 20–25% official success rate with high seed variance. INTACT + SADL reaches nearly 97% by epoch 5. Across eight demand regularizers (Gaussian, Student-t, torus, spherical shell, VICReg, etc.), INTACT + SADL exceeds 92% zero-shot control success after one epoch; seven of them hit 95.03–95.81%. Aligning backend and update counts, INTACT + SADL achieves 96.3% by epoch 2, with identical gains using MATH or Flash attention, confirming the improvement is not from backend optimizations.
Image Classification Results
SADL also accelerates convergence and stabilizes representations on ImageNet-LT, Galaxy10, CIFAR-10, and Imagenette. Attention maps shift from background to foreground (dog faces, galaxy cores). 3D feature geometry shows tight, separated class clusters versus loose, overlapping baseline clusters. Validation curves consistently stay above baselines. On Imagenette, SADL leads by 8.91 points at epoch 20, narrowing to 0.44 at epoch 800; it reaches the baseline's final value with 23% fewer updates. On ImageNet-LT, it reaches 90% of baseline final value with 10% fewer updates. SADL can even recover near-collapsed checkpoints: continuing from a collapsed point, INTACT recovers from 18.9% to 68.3% in one epoch.
Future Work
The authors plan to generalize the supply capacity concept to nonlinear, multimodal, and temporally structured transport operators.
Resources
Paper: https://arxiv.org/abs/2611.21596
Project page: https://zju3dv.github.io/SADL/
Code: https://github.com/zju3dv/SADL
Code example
👀点击关注
AI前沿快速FollowSigned-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
