Skip to content
VLMs Prioritize Object Size Over Humans in Scene Understanding, NeurIPS 2026 Finds

VLMs Prioritize Object Size Over Humans in Scene Understanding, NeurIPS 2026 Finds

A NeurIPS 2026 study reveals that vision-language models (VLMs) significantly differ from humans in scene understanding: VLMs over-rely on object size, centeredness, and low-level saliency, while humans naturally focus on people. The researchers introduce Counterfactual Semantic Saliency (CSS), a black-box evaluation method, and a dataset of 307 scenes with 1,306 counterfactual variants, showing all 19 tested VLMs fall short of human-level alignment, with size sensitivity being the primary driver of this gap.

Machine Heart
Machine Heart
Machine Heart

Background and Motivation

Vision-language models (VLMs) excel at scene understanding and visual question answering, but it remains unclear whether their internal scene representations align with human perception. Prior computer vision research has shown that models and humans can achieve similar task performance while relying on fundamentally different features, leading to divergent behaviors. Researchers from UC Santa Barbara investigate this gap for VLMs, asking whether models truly understand scene semantics the way humans do.

Contribution 1: Counterfactual Semantic Saliency (CSS) — A Black-Box Evaluation Framework

Why a Black-Box Method?

Modern VLMs have massive parameter counts and many are closed-source. Traditional white-box interpretability methods (attention maps, gradient heatmaps) are either inapplicable or fail to reveal the true causal features driving model outputs. The paper proposes Counterfactual Semantic Saliency (CSS) , a model-agnostic black-box method that quantifies each visual element's influence on the generated scene description through causal intervention.

CSS Mechanism

For each object in a scene, CSS removes that object (via inpainting or masking) and measures how much the model's description changes. If removing an object causes a large semantic shift, that object has high CSS; if removal barely changes the description, its CSS is low. This directly captures the causal contribution of each object to the model's scene understanding.

CSS method overview: remove object, measure description change
CSS method overview: remove object, measure description change

Contribution 2: Large-Scale Human Baseline Dataset

Dataset Construction

The researchers built a dataset on top of the widely used OSIE visual saliency dataset:

307 original scenes

1,306 counterfactual scenes (3–6 per original), each with one target object removed

Human Data Collection

Using the Prolific platform, they recruited 227 participants and employed a between-subjects design to avoid memory and carryover effects. For a scene with N target objects, the original and N counterfactual variants were split into N+1 non-overlapping stimulus sets, each completed by ~10 independent participants. This yielded 16,665 trials ; after removing ~0.3% low-quality responses, 16,603 scene descriptions remained (~10 per image), forming a robust human baseline.

Dataset statistics and between-subjects design illustration
Dataset statistics and between-subjects design illustration

Contribution 3: Systematic Human–VLM Gap Analysis

19 Models Evaluated, None Reach Human Level

The study evaluated 19 mainstream open-source and closed-source VLMs , including Qwen3-VL, InternVL3, Gemma 4, GPT-5.4, Claude 4.x, and Gemini 3.x series. Across two tasks — identifying the most semantically salient object and ranking object importance — all models showed statistically significant deviations from human consensus .

Model vs human performance comparison across 19 VLMs
Model vs human performance comparison across 19 VLMs

Four Hypotheses for the Gap

To explain why models diverge from humans, the authors tested four candidate factors drawn from human vision literature and known deep learning biases:

Sensitivity to Size — measured by segmentation mask area.

Sensitivity to Center — measured by negative distance from object center to image center.

Sensitivity to Low-Level Saliency — quantified using the Graph-Based Visual Saliency (GBVS) model, taking the peak saliency within each object region.

Sensitivity to Person/Non-Person — binary label (person vs. non-person) annotated by GPT-5.4.

Correlation analysis between each object's CSS and these four attributes revealed that VLMs are significantly more reliant on low-level visual properties (size, centeredness, low-level saliency) than human observers , while model sensitivity to the presence of people is significantly lower than humans' . In other words, models are easily distracted by large, central, visually salient objects regardless of semantic importance, whereas humans are strongly drawn to people in the scene.

Correlation of CSS with size, center, low-level saliency, and person sensitivity for humans vs. models
Correlation of CSS with size, center, low-level saliency, and person sensitivity for humans vs. models

Size Sensitivity Is the Primary Driver of the Gap

To determine which factor most causes models to misidentify the key object, the authors computed for each model two gaps: (1) Δr — the difference between the model's sensitivity to an attribute and human sensitivity; (2) ΔAcc — the difference between the model's Top-1 accuracy and human Top-1 accuracy. Across the 19 models, only the size-sensitivity gap (Δr for size) showed a significant correlation with the accuracy gap (ΔAcc) . The other three attributes did not reach significance. This explains the opening example: a large white bird occupies many more pixels than a fishing rod, so the model's over-reliance on size masks the rod's true importance, leading to a judgment opposite to humans.

White-Box Attention Maps Do Not Reveal True Causal Drivers

The researchers compared CSS with three common white-box interpretability methods on the open-source Qwen3-VL-4B-Instruct: raw attention maps, Grad-CAM, and AGCAM (Attention-Guided Class Activation Mapping). Because the model generates descriptions token by token, they aggregated heatmaps across all tokens (taking the maximum per token, then the maximum within each object region). The resulting white-box importance rankings were either noisy, overly sparse, or focused on semantically irrelevant regions, and often correlated negatively with CSS . This demonstrates that passive observation of internal activations fails to uncover the true causal factors driving VLM outputs in complex, modern architectures.

Comparison of CSS with white-box attention, Grad-CAM, and AGCAM on Qwen3-VL-4B-Instruct
Comparison of CSS with white-box attention, Grad-CAM, and AGCAM on Qwen3-VL-4B-Instruct

Conclusion and Implications

The paper introduces Counterfactual Semantic Saliency (CSS), a black-box, model-agnostic evaluation framework, and combines it with large-scale human psychophysics to systematically expose a pervasive gap between VLMs and humans in scene understanding: humans naturally attend to people, while models default to over-relying on object size, centeredness, and low-level saliency, with size sensitivity being the dominant factor . As synthetic model-generated data becomes increasingly used for training, the question "Do models see the world like humans?" deserves urgent attention.

Resources

Paper: https://arxiv.org/abs/2605.13047

Code: https://github.com/starsky77/Counterfactual-Semantic-Saliency-Data

Project page: https://starsky77.github.io/Counterfactual-Semantic-Saliency

Dataset: https://huggingface.co/datasets/starsky77/counterfactual-semantic-saliency

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

vision-language modelsScene UnderstandingCausal InterventionBlack-Box EvaluationNeurIPS 2026Counterfactual Semantic SaliencyHuman-Model AlignmentSize Bias
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.