Deep Pre-Alignment (DPA): Tsinghua’s New VLM Architecture Aligns Vision Before Language Understanding

Recent years have seen Vision‑Language Models (VLMs) become the dominant paradigm for multimodal understanding and reasoning. Most VLMs adopt a simple “ViT + lightweight projector + LLM” pipeline, feeding visual features directly into the LLM.

The authors identify a hidden problem: after projection, visual features are not naturally situated in the textual representation space familiar to the LLM.

To address this, they propose Deep Pre‑Alignment (DPA), which inserts a small perceiver VLM (containing a ViT, projector, and a shallow language module) before the target LLM. This module aligns visual features into a space closer to the LLM’s text embeddings, allowing the LLM to devote more capacity to high‑level understanding and reasoning.

Why DPA is needed – the “implicit alignment cost”

The LLM’s shallow layers must spend considerable capacity on coarse visual‑to‑text alignment.

Model depth that could be used for deep reasoning is consumed early.

Multimodal training can damage the LLM’s original language ability, causing forgetting.

DPA moves the alignment step ahead of the LLM, relieving these pressures.

Core characteristics of DPA

From “ViT directly to LLM” to “deep pre‑alignment before LLM”

Traditional VLMs feed projected visual tokens straight to the LLM, relying on the LLM’s early layers to adapt them. DPA replaces the ViT encoder with a perceiver VLM that includes a language‑aware module, producing pre‑aligned multimodal representations for the LLM.

No changes to training or inference pipelines

DPA does not require extra loss terms, special optimization targets, or new inference steps. It follows the common two‑stage VLM training:

Stage‑1: Train the projector on image‑text pairs to align dimensions between the perceiver VLM and the target LLM.

Stage‑2: Fine‑tune the entire system on high‑quality visual‑instruction data.

Thus DPA can be inserted by swapping the original ViT with the perceiver VLM while keeping training and inference unchanged.

Significant multimodal performance gains

The authors evaluated DPA on 11 benchmarks (8 multimodal tasks and 3 text tasks). Results include:

Qwen3‑4B: +1.9 average points across 8 multimodal benchmarks.

Qwen3‑32B: +3.0 points.

LLaMA‑3.2‑3B as LLM: +3.6 points, showing the benefit is model‑family agnostic.

Mitigates language‑ability forgetting

On the MATH‑500 test, a standard LLaVA‑NeXT style model drops from 84.8 to 36.4 after multimodal training. With DPA the score recovers to 54.2, reducing forgetting by 32.9 % for 4B models and 21.6 % for 32B models.

Minimal inference overhead

For a 32B model, DPA adds only about 2 % throughput reduction while delivering stronger multimodal performance and better language retention.

Why DPA works – ablation evidence

Controlled experiments show the gains are not solely due to a stronger visual module. Even an untrained perceiver VLM yields ~+3.5 average points over the baseline. Removing the language component of the perceiver VLM drops average performance from 49.6 → 50.3, while the full DPA reaches 53.0.

Eliminating pre‑training of the perceiver’s language model further degrades performance to 38.7, confirming that the language‑aware alignment is the key factor.

Conclusion

DPA proposes a new VLM architecture where visual features are pre‑aligned to the LLM’s textual space before entering the LLM. It improves multimodal understanding, preserves language ability, and incurs negligible extra inference cost, demonstrating that high‑performance VLMs require not only stronger vision encoders but also better alignment to textual representations.

Paper: https://arxiv.org/abs/2605.15300
GitHub: https://github.com/THUMAI-Lab/Deep-Pre-Alignment