RAVEL: Two Simple Tweaks Double Vision Encoder Evaluation Correlation for MLLMs

Introduction

Vision encoders are core components of multimodal large language models (MLLMs). Even with the same LLM, different vision encoders can yield significantly different downstream performance. The most direct way to find the best encoder is to plug each candidate into the LLM, complete cross-modal alignment training, and evaluate — but this requires massive compute. The key research question: how to accurately predict a vision encoder's downstream performance before full training?

Limitations of Existing Evaluation Methods

Existing methods fall into two categories: unimodal methods that assess the vision encoder alone (e.g., image classification accuracy), and cross-modal methods that also consider the target LLM by measuring vision-language representation relationships. Intuitively, cross-modal methods should be more accurate because they account for vision-language matching.

However, the team trained and evaluated 210 MLLMs and discovered that cross-modal methods' advantage is unstable. In small, homogeneous candidate pools, cross-modal methods outperform unimodal ones; but as the pool grows in size and diversity (different architectures, pretraining strategies, input resolutions, continuous and discrete tokenizers), cross-modal performance drops sharply, sometimes falling below unimodal methods. This reveals a clear limitation of current cross-modal metrics for large-scale, diverse vision encoder selection.

RAVEL: Cross-Modal Nearest-Neighbor Consistency

RAVEL adopts a cross-modal nearest-neighbor consistency framework. Core intuition: if a candidate vision encoder and the target LLM form similar semantic structures on the same samples, then the nearest-neighbor relationships in their respective representation spaces should be consistent.

Given a batch of images and corresponding texts, RAVEL extracts features using the candidate vision encoder and the target LLM separately. In each representation space it retrieves nearest neighbors: images find similar images, texts find similar texts. For an image-text pair, if the image-side neighbors' corresponding texts also appear in the text-side neighbors, the two spaces share the same semantic neighbors. More such overlaps indicate higher local structural consistency between vision and language representations.

RAVEL aggregates nearest-neighbor consistency across all samples to score the candidate vision encoder. Because it only compares internal nearest-neighbor relationships within each space, it does not require vision and text features to have the same dimensionality, nor any training or downstream performance labels.

Two Key Improvements

1. Stabilized PCA Whitening

Different models' representation spaces often have different feature distributions: a few high-variance directions can dominate similarity computation, and dimensions may be strongly correlated, causing retrieved neighbors to poorly reflect true semantic relationships. PCA whitening mitigates this, but standard whitening over-amplifies low-variance directions. RAVEL adds a stabilization term to suppress this amplification, followed by L2 normalization. Further analysis shows the performance gain comes mainly from the stabilization term, with L2 normalization providing additional improvement.

2. Patch-Level Similarity Scoring

Two images may contain similar objects but differ in viewpoint, object position, and background. Global pooling mixes all this information, diluting the object-level similarity. RAVEL directly matches local image patches: each patch in one image finds its most similar region in the other image, and bidirectional matching results are aggregated without requiring pre-established one-to-one correspondences. Visualization shows high-similarity matches concentrate on semantically related regions, while unrelated regions receive low scores.

Large-Scale Experiments: 210 MLLMs

The team combined 70 vision encoder configurations with three LLMs — Qwen3-1.7B, Qwen2.5-1.5B, and SmolLM2-1.7B — training 210 MLLMs total. All models used a unified LLaVA-style two-stage training pipeline and were evaluated on 11 downstream benchmarks covering visual question answering, multimodal reasoning, and text recognition.

RAVEL was compared against nine existing evaluation metrics, including unimodal metrics and cross-modal methods with and without extra training. For fairness, all methods were compared under similar FLOPs budgets. Results: on all three LLMs, RAVEL achieved the highest Spearman and Pearson correlation coefficients. Average Spearman correlation reached 0.85 , surpassing the training-required Alignment Probing method by +36.1% , and the prior training-free methods GW by +93.5% and MutualNN by +101.1% .

Ablation Study: Why RAVEL Works

Ablation experiments confirm that RAVEL's gains come from both designs: PCA whitening significantly improves nearest-neighbor quality, and patch-level matching further boosts predictive power on top of that.

Open Resources

Paper: https://arxiv.org/abs/2610.05413 Code: https://github.com/JuntaoTang/MLLM-VisionEncoder-Eval Model weights (210 MLLMs):

https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo

The open-sourced 210 MLLM weights cover diverse vision encoder–LLM combinations, enabling researchers to directly conduct vision encoder selection, cross-modal alignment, and representation analysis studies without repeating large-scale multimodal training.

Conclusion

This work systematically investigates the practical but under-verified problem of selecting suitable vision encoders for target LLMs before full MLLM training. By training and evaluating 210 MLLMs, the team found that existing cross-modal metrics do not consistently maintain advantages in complex candidate pools. RAVEL addresses this by measuring cross-modal nearest-neighbor consistency, enhanced with stabilized PCA whitening and patch-level matching, achieving reliable downstream performance prediction without extra training. The released code and model weights provide a strong baseline and reusable resources for vision encoder selection, cross-modal alignment, and multimodal representation research.