AIWalker
May 19, 2026 · Artificial Intelligence
Why Attention Transfer Fails for DINOv2 and Other Modern ViTs: Architecture Mismatch Revealed
A large-scale benchmark of 20 pretrained ViT teachers across 11 families shows that attention copy and distillation improve some models but hurt others—especially DINOv2, CLIP, and BEiTv2—due to architecture mismatches, and adding the teachers' native components to students restores the lost performance.
Architecture CompatibilityAttention TransferDeep Learning
0 likes · 13 min read