Why Multimodal Video Models Still Miss the Mark: Inside the New Video‑MME‑v2 Benchmark

Background and Motivation

When users watch videos with multimodal large models, they often feel the model knows a little about everything but fails to answer specific questions accurately. Existing video‑understanding leaderboards show high scores, yet real‑world performance is disappointing. Video‑MME‑v2 introduces a brand‑new non‑linear scoring mechanism to bring model evaluation back to reality.

Benchmark Design

Video‑MME‑v2, released in 2024, evaluates models under varying video‑length conditions and has become a standard test set for many large models, including Gemini and GPT. The benchmark structures its assessment into three progressive layers:

Layer 1 – Multi‑point Information Aggregation (C1): Tests the model’s ability to retrieve and extract dispersed clues from video frames, audio, and subtitles.

Layer 2 – Temporal Understanding (C2): Requires the model to parse state changes, action sequences, and event logic across time.

Layer 3 – Complex Temporal Reasoning (C3): Demands the model to combine multimodal temporal cues with world knowledge and commonsense to solve multi‑step reasoning tasks.

The system classifies questions into these layers, allowing precise identification of a model’s capability gaps.

Scoring Mechanism

Traditional benchmarks score each question independently, which lets models guess correctly by chance. Video‑MME‑v2 abandons this approach and adopts a grouped evaluation:

Consistency Groups: Four related questions probe a single ability (e.g., counting athletes, identifying actions, counting repetitions, total segment count). The model’s correct answer count N is transformed into a score (N/4)², rewarding only perfect consistency.

Reasoning‑Coherence Groups: A logical interrogation chain where a single mistake triggers a “first‑error‑cutoff,” nullifying any later correct answers.

This non‑linear scoring dramatically reduces inflated scores and highlights genuine competence.

Dataset Construction

Creating the 800‑video dataset required about 3,300 human hours. Over 80% of videos were released after 2025, with nearly 40% after October 2025. The team filtered out popular movies and top‑creator content to prevent models from exploiting memorized data. The videos span four domains—sports, entertainment, arts, education—covering 31 sub‑categories, averaging 10.4 minutes (53% under 10 minutes). Quality is high: 84.3% of videos have over 10 k views, averaging 4.83 million views.

Annotation involved 12 human experts who designed questions and crafted eight answer options per question, including deceptive distractors. Afterwards, 50 independent experts performed blind cross‑testing; any question solvable without watching the video was discarded.

Results and Analysis

Under the traditional average accuracy metric, Gemini‑3‑Pro and Gemini‑3‑Flash score 66.1% and 61.1% respectively. When evaluated with the non‑linear score, their results drop to 49.4% and 42.5%, exposing a large performance gap.

Model robustness is reflected by the ratio of non‑linear score to average accuracy. Smaller models like LLaVA‑Video‑7B achieve only about 40% of the ratio, indicating frequent random hits. In consistency groups, strong models maintain stable accuracy, but in reasoning‑coherence groups accuracy declines steadily as tasks require deeper causal inference.

Subtitle presence markedly improves performance: models gain a stable boost when textual cues are available, while pure visual reasoning suffers severe degradation, revealing a heavy reliance on explicit semantic signals.

Capability Abstraction

The benchmark abstracts model abilities into three blocks: full‑modal information aggregation (C1), long‑context understanding (C2), and complex reasoning (C3). Models excelling in all three dominate the leaderboard, though large parameter counts can partially compensate for missing capabilities. For example, Qwen3.5‑397B‑A17B‑Think, despite lacking explicit full‑modal design, scores 39.1 points thanks to its massive scale.

Frame‑count also matters: Qwen3.5‑397B processes 512 frames and scores 8.5 points higher than with 64 frames, showing that longer context yields deeper video comprehension.

Conclusion

Even the most advanced AI models appear competent on traditional metrics, but Video‑MME‑v2’s rigorous evaluation uncovers their true limits. In coherent video‑logic reasoning, they still behave like hesitant apprentices, indicating substantial challenges remain on the path toward general artificial intelligence.