NeurIPS 2026: DGP Enables Dynamic Layer-Wise Vision-Language Routing Across All ViT and LLM Layers

Problem: Fixed Visual Projection Limits Multimodal Reasoning

Current multimodal large language models (MLLMs) typically extract visual features from a single ViT layer (often the last or penultimate), project them once, and feed them into the LLM input. All subsequent LLM layers must reason from this fixed visual representation, regardless of whether the task requires fine-grained texture details (e.g., reading menu prices) or high-level semantics (e.g., interpreting artwork). The article illustrates this with a menu example: LLaVA-1.5 misreads prices and hallucinates an extra calzone, while DGP correctly reads 6.90 and 7.90 euros and computes 14.80 euros.

Core Idea: Dynamic Gated Pathways (DGP)

DGP re-frames the connection between vision and language as a learnable routing problem: which ViT layer should connect to which LLM layer , conditioned on the current image-question context and the semantic state of each language layer. It opens all ViT–LLM layer pairs as candidates and lets the model dynamically assign fusion weights.

2.1 Glimpse Token as Query

A learnable glimpse token is inserted after the image and question tokens but before the first answer token. Due to causal attention, it sees the full image-text context. Its hidden state at each LLM layer serves as a query signal representing what that layer currently seeks.

2.2 Dynamic Weight Allocation via Shared Cross-Attention and Layer-Specific Routers

Shared multi-head cross-attention compares the glimpse query against [CLS] summaries of all ViT layers (mapped to LLM hidden space), producing a unified cross-modal routing feature.

Layer-specific routing MLPs (one per LLM layer) then apply Softmax to generate a weight distribution over all ViT layers.

This design allows different LLM layers to develop distinct visual preferences, and the same layer to route differently across samples and tasks.

2.3 Position-Aligned Fusion and Gated Residual Injection

For each LLM layer, the routed ViT features are fused at the same patch positions :

Fused patch = Σ (weight_k × patch_k at that position)

The fused result passes through a layer-specific projector and a learnable scalar gate, then is added residually to the layer's visual tokens. Thus visual information continuously participates in every LLM layer's representation update.

2.4 Entropy Regularization to Prevent Premature Collapse

During pre-training, an entropy regularization term (coefficient 0.01) maximizes the average routing distribution entropy across a batch, encouraging exploration of all ViT layers. The regularizer is turned off during instruction tuning to allow specialization.

Experimental Results

Main Benchmarks (LLaVA-1.5-7B)

DGP improves all 11 reported metrics over the LLaVA-1.5 baseline:

SQA-I: 66.8 → 71.0 (+4.2)

MM-Vet: 30.5 → 33.8 (+3.3)

VizWiz: 50.0 → 53.1 (+3.1)

LLaVA-Wild: 63.4 → 66.5 (+3.1)

On 13B, 10 of 11 metrics improve (VizWiz +3.8, MMBench +2.4, GQA +2.2). DGP also outperforms other multi-layer visual feature methods on 6 of 7 benchmarks at 7B.

Cross-Architecture Generalization

DGP is plugged into two distinct frameworks without modifying their core designs:

Qwen3-VL-2B (SigLIP2-L + Qwen3-1.7B): SQA-I 85.3 → 86.9

LLaVA-NeXT (CLIP-L + Vicuna-7B, AnyRes): SQA-I 69.5 → 72.3, MMBench 66.5 → 68.7

All four reported metrics improve, indicating transferability across visual encoders, language backbones, and resolution strategies.

Compute Overhead

At batch size 1:

7B: +5.29% parameters, +4.07% FLOPs, peak memory ~unchanged

13B: +4.27% parameters, FLOPs ~flat, peak memory +4.13%

Key Ablation: Dynamic Routing vs. Naive Averaging

Averaging all 24 ViT layers uniformly degrades performance significantly (MM-Vet 30.5 → 25.3, MMBench 64.3 → 57.6, LLaVA-Wild 63.4 → 56.3). Adding dynamic gated routing recovers and surpasses baseline (MM-Vet 32.4, MMBench 65.6). Entropy regularization further boosts (MM-Vet 33.8, MMBench 66.7, SQA-I 71.0). This confirms that learned layer weights, not merely more layers, drive gains .

Routing Visualization: Dynamic and Interpretable

Across LLM layers: Middle ViT layers are most activated, but shallow and deep layers are selected by specific language layers; weight curves show multiple peaks, indicating combination of granularities.

Across tasks: Fixed LLM layer 21 shows different ViT weight distributions for LLaVA-Wild, VizWiz, and SQA-I.

Across samples: Routing varies per input, not just per dataset.

Fixed image, varying question: Color/detail questions shift weight to shallow layers; semantic/holistic questions shift to deep layers; "describe in detail" spreads weight across many layers.

Spatial feature maps confirm DGP produces sharper object contours and preserves fine structures (spoon edges, building boundaries) compared to fixed-layer or averaged features.

Limitations and Outlook

Current validation focuses on LLaVA-1.5 (7B/13B), Qwen3-VL-2B, and LLaVA-NeXT. Larger language models (>30B), more visual encoders, and larger-scale multimodal corpora remain untested. Compute overhead, while modest now, may grow with layer count and resolution. The conclusion is not that DGP solves the vision bottleneck, but that the vision-language wiring scheme is a critical, under-explored design dimension worthy of further investigation.