ICML 2026 Spotlight: MetaphorVU – The First Benchmark for Metaphorical Video Understanding

Background

In short‑video and social‑media contexts, creators increasingly convey deep meanings through visual metaphors—for example, a scene with tuxedo‑clad pigs feasting at a banquet while a cat scrambles for leftovers symbolizes elite exploitation and grassroots struggle. Humans grasp such metaphors instinctively, but multimodal large language models (MLLMs) can recognize objects yet fail to map visual elements to abstract concepts, a gap termed cross‑domain mapping , which serves as a litmus test for high‑order cognition.

Benchmark Construction – MetaphorVU‑Bench

To measure “does the model truly understand metaphors?”, the team built MetaphorVU‑Bench , the first comprehensive benchmark with three key properties:

Systematic classification based on multimodal metaphor theory, covering eight categories (Body Language, Atmosphere Language, Cultural Symbol, Naturalistic Symbol, Causal Montage, Analogical Montage, Surreal Narrative, Performative Narrative).

Data sourced from billions of real short videos, ensuring topic diversity and compatibility with existing MLLM input lengths.

Strict multi‑stage funnel filtering: videos with >150 comments (~70 k) → GPT‑5 summarization/ASR (~16 k) → Gemini‑3‑Pro visual consistency check (~4 k) → manual balancing, yielding 860 high‑quality metaphor videos.

Each video is paired with a free‑text answer; scoring uses LLM‑based evaluation on a 0–100 scale, following free‑text video‑QA practice.

Defect Analysis of Existing Models

Using the benchmark, 11 representative closed‑source and open‑source MLLMs were evaluated. Even the strongest models (Gemini‑3‑Pro, GPT‑5) achieved average scores around 64, far below the human upper bound of 83.4, a ~20‑point deficit. Some reasoning‑enhancement methods (e.g., LTR, ViTCoT) actually degraded performance, indicating that metaphor understanding requires capabilities distinct from object recognition or event description.

Fine‑grained error analysis categorized failures into four types: recognition errors, missing mappings, surface mappings, and incorrect mappings. Over 80 % of failures were due to missing or wrong cross‑domain mappings, especially in categories demanding strong mapping (Causal Montage, Surreal Narrative).

MetaphorBoost – Mapping Enhancement

To address the identified shortfall, the authors built the first metaphor‑specific knowledge graph (≈54 k nodes, 200 k edges) by extracting source‑target concept pairs from public metaphor datasets using DeepSeek‑V3.2 and linking them into a graph.

MetaphorBoost augments inference in three steps:

Identify : the MLLM extracts visual keywords from the video.

KG Querying : keywords seed multi‑hop retrieval in the metaphor KG; top‑z concepts most connected to the keywords are selected.

Generate : retrieved concepts guide the model’s reasoning to produce a metaphor interpretation.

This external cognitive scaffold improves the model’s ability to link visual elements to underlying concepts without retraining.

Experimental Results

MetaphorBoost yields consistent gains across models:

Gemini‑3‑Pro + 2.3 points

Qwen2.5‑VL‑7B + 4.1 points

Qwen3‑VL‑8B‑Thinking + 3.8 points

Improvements are especially pronounced on mapping‑heavy sub‑categories (up to +7.8 points). Ablation studies confirm that removing the external KG, collapsing the graph to plain text retrieval, or substituting ConceptNet all degrade performance, demonstrating the necessity of a dedicated metaphor KG.

Further analysis shows simultaneous reductions in missing, surface, and incorrect mappings, directly proving that MetaphorBoost strengthens the visual‑element‑to‑concept link rather than merely adding prompts.

Significance and Outlook

MetaphorVU‑Bench provides the first systematic metric for evaluating high‑order multimodal cognition, shifting assessment from “can the model see?” to “can the model understand?”. The cross‑domain mapping diagnosis pinpoints the cognitive bottleneck, and MetaphorBoost offers a plug‑and‑play enhancement path, validating the feasibility of external knowledge‑graph‑driven mapping improvement. Future work will explore stronger mapping models and broader metaphor scenarios to advance multimodal AI from perceptual to cognitive intelligence.