Spectral Disentanglement and Enhancement: Teaching Multimodal Models to Denoise and Purify

Introduction

Large‑scale multimodal contrastive learning has achieved impressive representation quality, yet it treats all feature dimensions uniformly and ignores the intrinsic spectral structure of learned embeddings. Empirical studies show that high‑dimensional embeddings often collapse into a narrow cone, with task‑relevant semantics confined to a small subspace while the majority of dimensions are occupied by noise and spurious correlations, severely harming generalization.

Problem Statement

The authors identify two fundamental issues:

Spectral imbalance: Core semantic features concentrate in a limited subspace, while weak signals and noise dominate the remaining dimensions.

Uniform optimization flaw: Standard contrastive objectives weight every dimension equally, leading to semantic entanglement, robustness loss, and over‑emphasis of false correlations.

Spectral Disentanglement

Given a feature matrix F \in \mathbb{R}^{m\times n} extracted from a visual‑language model (VLM), the authors apply singular value decomposition (SVD) F = U \Sigma V^{\top}. The singular values \sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0 quantify the energy along each orthogonal basis vector in V. Small singular values correspond to redundant or noisy components, while large singular values encode the most informative, task‑relevant directions.

Using the inter‑quartile range (IQR) and the Marchenko‑Pastur distribution as theoretical bounds for random covariance matrices, the authors partition the feature dimensions into three subspaces based on singular‑value magnitude:

Strong signal: Dominant dimensions containing core semantics.

Weak signal: Subtle but potentially useful variations.

Noise: Random fluctuations that degrade robustness.

Spectral Enhancement

After disentanglement, the framework applies a curriculum‑driven enhancement matrix \Delta that treats each subspace differently:

Strong‑signal amplification: Inject controlled adversarial noise whose intensity is modulated by training progress \alpha(t) and the relative size of the singular value, preventing over‑fitting while strengthening discriminative features.

Weak‑signal normalization: Scale down weak singular values adaptively with \alpha(t) to preserve useful fine‑grained information without destabilizing training.

Noise suppression: Apply signal‑to‑noise‑ratio regularization that aggressively attenuates noise singular values early in training and relaxes the penalty as the model stabilizes.

The reconstructed enhanced feature matrix is F' = U (\Sigma + \Delta) V^{\top}, where \Delta is diagonal and respects the three subspace strategies. The Frobenius norm of \Delta is bounded, guaranteeing training stability.

Dual‑Domain Contrastive Learning

To fully exploit the benefits of spectral disentanglement, the authors propose a dual‑domain loss that aligns representations both in the instance space and in the spectral space.

Instance‑level alignment: Standard InfoNCE loss<br>

Spectral distribution alignment: Align singular‑value vectors \sigma of paired modalities using Hellinger distance, ensuring consistent importance weighting across modalities.

Subspace consistency: Enforce orthogonal alignment of the top‑ k singular vectors V_{1:k} by minimizing the deviation of their Gram matrices, preventing arbitrary rotations that preserve instance distances but destroy semantic structure.

The overall objective combines these terms with a dynamic weight \lambda(t) that gradually shifts emphasis from global spectral regularization to fine‑grained instance alignment, avoiding excessive regularization.

Experiments

Evaluation is performed on the MMEB benchmark, which aggregates 36 multimodal datasets covering image classification, visual question answering, retrieval, and localization. The authors report Precision@1 for each dataset.

Key findings:

The SDE framework consistently outperforms all baselines on both in‑distribution and out‑of‑distribution test sets.

When trained on a single task (e.g., retrieval), the model exhibits strong cross‑task transfer, improving classification and localization performance by 27% and 17% over the VLM2Vec baseline.

Qualitative analysis shows that the proportion of strong‑signal energy rises from 10.42% to 17.32%, weak‑signal from 7.75% to 12.17%, while noise drops from 81.84% to 70.51%, concentrating 88.7% of semantic energy in just 17% of dimensions.

Conclusion

The Spectral Disentanglement and Enhancement (SDE) framework establishes a theoretical link between embedding geometry and spectral characteristics, introduces adaptive spectral processing, and integrates a dual‑domain contrastive loss. Empirical results demonstrate that SDE markedly improves robustness and generalization of multimodal representations, offering a practical and generalizable enhancement to existing contrastive learning pipelines.