Why Sharing Parameters in Vision Transformers Hurts Performance—and How Layer Specialization Fixes It

Background and Motivation

Vision Transformers (ViT) split an image into patches and prepend a special [CLS] token before feeding all tokens into identical Transformer blocks. Although [CLS] captures global semantics and patches capture local details, both share the same linear layers, normalization layers, and parameters, leading to an internal conflict.

Meta FAIR researchers discovered that this "one‑size‑fits‑all" treatment creates significant computational friction. To alleviate it, they propose a simple strategy called Layer Specialization , which breaks parameter sharing between [CLS] and patch tokens.

Method Details: Customized Paths

Layer Specialization assigns dedicated computation paths for [CLS] and patches in selected Transformer layers:

Normalization Layer Specialization : Use two independent LayerNorm parameters—one for [CLS], another for patches.

QKV Projection Specialization : Before attention, project [CLS] with a dedicated weight matrix while patches use a separate matrix.

Unified Interaction Space : Despite separate projections, the resulting Q, K, V vectors are still pooled together, allowing [CLS] to attend to all patches.

LayerScale Specialization : Provide distinct scaling factors for the two token types.

The design keeps the overall Transformer architecture unchanged, so inference FLOPs remain identical to the standard ViT.

Where Specialization Matters Most

Normalization layers are the cost‑effective sweet spot : Specializing only LayerNorm adds less than 0.1% parameters but yields noticeable gains.

The first third of Transformer blocks are the “golden zone” : Applying QKV specialization to the early layers provides the best performance boost.

Zero inference overhead : Training adds about 8% more parameters, yet each token still passes through a single linear transformation per layer, keeping FLOPs unchanged.

Experimental Results

Both self‑supervised (DINOv2) and fully supervised (DeiT‑III) settings were evaluated. On ViT‑L, Layer Specialization improves dense prediction tasks:

Semantic Segmentation : ADE20K mIoU rises from 46.2 to 48.4 (+2.2); Cityscapes from 65.2 to 67.4 (+2.2).

Depth Estimation : RMSE on KITTI and NYU‑Depth v2 drops significantly, yielding sharper edge details.

Object Detection : COCO AP increases by roughly 2.2 points.

Training curves show faster convergence and higher final accuracy for the specialized model.

Qualitative Insights

PCA visualizations reveal that standard DINOv2 often produces patch‑CLS conflicts, manifesting as bright artifacts. After layer specialization, feature maps become smoother, semantic boundaries clearer, and background noise is suppressed.

Lightweight Extension with LoRA

To further reduce parameter growth, the authors explore Low‑Rank Adaptation (LoRA). By representing the specialized weight matrices as A·B with low rank, the extra parameters stay below 1% while preserving performance gains.

Conclusion

Layer Specialization challenges the prevailing belief in uniform ViT architectures. By giving [CLS] and patch tokens their own dedicated parameters—especially in early layers and normalization stages—models achieve cleaner feature representations and notable improvements on dense prediction benchmarks, all without any inference cost.