Google DeepMind Open‑Sources TIPSv2: State‑of‑the‑Art Patch‑Text Alignment at CVPR 2026

In the fast‑moving field of multimodal large models, dense image‑text alignment—matching each image patch to its textual concept—remains a critical bottleneck. DeepMind’s new paper, TIPSv2: Advancing Vision‑Language Pretraining with Enhanced Patch‑Text Alignment , addresses this gap and was accepted at CVPR 2026.

The authors first observed a counter‑intuitive phenomenon: a small patch‑level student model distilled from a larger teacher outperforms the teacher on dense tasks such as zero‑shot segmentation. Further analysis revealed that the key difference lies in how “visible patches” are supervised. Traditional masked image modeling (e.g., iBOT) computes loss only on masked tokens, whereas TIPSv2’s distillation provides the student with rich features for all patches, unlocking superior dense alignment.

Three Core Technical Innovations

1. iBOT++ – Global‑Perspective Self‑Supervised Alignment Engine : Extends the classic iBOT loss from masked tokens to all tokens, forcing the model to maintain fine‑grained consistency across the entire image. This change alone raises zero‑shot segmentation mIoU on ADE20K from 3.5 % to 17.6 % (+14.1 %).

2. Head‑only EMA – Memory‑Efficient Exponential Moving Average : Instead of applying EMA to the full model (which would exhaust GPU memory at billion‑parameter scale), only the projection heads are updated with EMA while the visual backbone’s EMA is frozen. This preserves performance while dramatically reducing memory consumption.

3. Multi‑Granularity Captions : To prevent shortcut learning on coarse visual keywords, the training data combines traditional alt‑text with dense local subtitles generated by PaliGemma and richer global descriptions from Gemini Flash. Randomly alternating these captions during training yields stronger robustness on both dense alignment and global image‑text retrieval.

Comprehensive Evaluation

TIPSv2 was evaluated on nine core tasks spanning dense image‑text (zero‑shot segmentation), global image‑text (classification, cross‑modal retrieval), and pure vision (semantic segmentation, depth estimation, surface normal prediction). Across twenty benchmark datasets, four model sizes (86 M – 1.1 B parameters) were tested. Results include:

Dominance on zero‑shot segmentation, surpassing SigLIP2, SILC, and DINOv2.

First or second place on five of seven global tasks.

First or second place on seven of nine pure‑vision tasks.

On a shared six‑task suite, TIPSv2 wins four, demonstrating the advantage of joint image‑text supervision over vision‑only pre‑training.

In a head‑to‑head comparison with the recent DINOv3 model (ViT‑L backbone), DINOv3’s teacher has six times more parameters and fifteen times more image data, yet TIPSv2 still outperforms it on four of six overlapping evaluations.

Feature Visualization

Principal component analysis of feature maps shows two major improvements over prior TIPS and SigLIP2: (1) far smoother representations that suppress background noise, and (2) sharper semantic focus with finer‑grained boundary detail, indicating deeper spatial understanding without manual labels.

Open‑Source Ecosystem

The release includes full model weights for B/14 (86 M), L/14 (303 M), SO400m/14 (412 M), and g/14 (1.1 B) in both PyTorch and JAX/Scenic, DPT prediction heads for depth, normal, and segmentation, and extensive Colab notebooks for feature visualization and zero‑shot segmentation. All code and models are licensed under Apache 2.0, facilitating both academic research and industrial deployment.

Overall, TIPSv2 demonstrates that fine‑grained patch‑level supervision combined with efficient training tricks can dramatically boost multimodal understanding, pointing toward a promising path for future AGI‑level vision‑language systems.