The Convergence of NLP and Computer Vision: Unified Neural Architectures and Pre‑training Strategies

Introduction – The transcript is the written version of a live presentation by Hu Han, Principal Research Manager at Microsoft, delivered at DataFunCon on December 17, 2022. It outlines three main topics: the unification trend of NLP and CV, neural‑architecture fusion, and pre‑training fusion.

1. NLP and CV Unification

Recent years have seen a convergence of natural‑language processing (NLP) and computer‑vision (CV) in both neural‑network architecture and pre‑training methods. Transformers, originally dominant in NLP, have become the mainstream architecture in CV (e.g., ViT, Swin Transformer), replacing convolutional networks.

Pre‑training techniques such as masked language/image modeling (MLM/MIM) are also converging. Methods like BERT/GPT in NLP and BEiT/MAE/SimMIM in CV share the same self‑supervised objective of predicting masked regions.

Brain‑science insights suggest that human intelligence relies on a unified predictive mechanism across modalities, with the thalamus playing a key role in visual prediction.

2. Practical Benefits of Unification

Technology and knowledge sharing accelerate progress across fields.

Facilitates multimodal applications such as CLIP and DALL‑E.

Reduces cost and improves efficiency; for example, modern GPUs (e.g., Nvidia H100) achieve up to 6× speed‑up for transformer training.

3. AI Neural‑Architecture Unification

Early AI models were domain‑specific (CNN for CV, Transformer for NLP, GNN for graphs). The trend is to apply successful architectures across domains: convolutional ideas to NLP (ConvSeq2seq) and transformer ideas to CV (ViT, Swin).

The Swin Transformer, proposed by the author’s group, demonstrates that a transformer‑based backbone can replace CNNs in tasks such as semantic segmentation and object detection, achieving state‑of‑the‑art performance.

Key contributions of Swin Transformer:

Integration of transformer modules with vision‑specific priors.

Shifted‑window design that reduces the number of windows and enables efficient parallel computation.

Training tricks and open‑source release that accelerated community adoption.

4. Swin Transformer V2

Building on the original Swin, Swin‑V2 introduces a new masked‑image‑modeling (MIM) pre‑training method that is self‑supervised and data‑efficient, allowing large‑scale dense vision models to be trained with far fewer labeled images. It also adds continuous relative‑position bias to improve resolution transferability and stabilizes attention computation.

The V2 model (≈30 B parameters) sets new records on four major benchmarks (object detection, semantic segmentation, video classification, image classification) while using 25 % fewer parameters and 40 × less annotation data compared with contemporaries.

5. Pre‑training Fusion and Scaling Laws

Mask‑image‑modeling (MIM) pre‑training follows a simple pipeline: random large‑patch masking, a lightweight linear prediction head, and pixel‑level reconstruction. Experiments show that larger models benefit more from this pre‑training, and the average distance (AvgDist) between visible and masked patches correlates with downstream performance.

Scaling laws observed in NLP (performance grows linearly with compute, data, and parameters) also hold for vision MIM, though the exact data‑size relationship remains less clear.

Additional insights include feature‑distillation techniques that transfer knowledge from large pre‑trained models to smaller student networks, improving fine‑tuning performance across tasks.

Conclusion – The talk emphasizes that AI tasks are moving toward a unified architecture and pre‑training paradigm, with transformer‑based models and masked‑modeling techniques bridging the gap between NLP and CV. Continued advances, especially in large‑scale vision models, are expected to follow the rapid progress seen in language models like ChatGPT.