Skip to content
VideoMemBench: Claude Opus 5 Leads 25 Models in Video Conversation Memory Test

VideoMemBench: Claude Opus 5 Leads 25 Models in Video Conversation Memory Test

VideoMemBench evaluates how well multimodal large language models recall and use prior dialogue context in video conversations, testing 25 models on episodic and answer dependencies; Claude Opus 5 scores 45.7 overall but answer dependency remains only 35.7%, revealing struggles with long-range dependencies and error propagation.

Machine Heart
Machine Heart
Machine Heart

VideoMemBench is a new benchmark designed to assess whether multimodal large language models (MLLMs) can remember and correctly use information established earlier in a video-grounded conversation. The benchmark was introduced by researchers from the University of Technology Sydney, the Chinese University of Hong Kong, the Hong Kong University of Science and Technology, and Zhejiang University.

Motivation: Natural Follow‑Up Questions Require Memory

The article illustrates the need with two everyday scenarios:

Basketball game: After identifying a player who scored, a later question asks for that player’s total points. The model must retrieve the player’s name from its own earlier answer and combine it with game statistics.

Movie discussion: After talking about an opening scene, a later question asks whether a character currently on screen appeared in that initial scene. The model must recall which scene was discussed first and check the character’s presence.

Both examples require linking the current question to prior dialogue — either to a previously mentioned scene (episodic dependency) or to a previously given answer (answer dependency).

Benchmark Design: Two Dependency Types

VideoMemBench organizes questions into multi‑turn dialogues around the same video, explicitly creating two kinds of dependencies:

Episodic Dependency — the question refers back to a specific scene discussed earlier; the model must identify that scene from the conversation history and then answer using the video.

Answer Dependency — the question builds on an entity, object, or event the model itself identified in a previous turn; the model must chain its own prior answers to answer correctly.

Figure 2: Multi‑turn QA linked by episodic and answer dependencies
Figure 2: Multi‑turn QA linked by episodic and answer dependencies

Data Construction: Planner–Generator–Judger Pipeline with Human Verification

To ensure each dependent question truly requires history, the team used three automated roles:

Planner — designs the dependency structure (which prior turns the new question relies on).

Generator — writes the actual question and reference answer.

Judger — checks that removing the relevant history makes the question unanswerable, that the question does not restate the needed prior information, and that the reference answer is grounded in the video.

Human experts then reviewed every retained dialogue. The final dataset contains 326 videos and 2,476 QA turns , of which 1,387 turns depend on prior context. Dependencies span multiple turns and some questions require multiple historical links simultaneously.

Figure 3: Data construction pipeline
Figure 3: Data construction pipeline
Figure 4: Dataset scale and dependency distribution
Figure 4: Dataset scale and dependency distribution

Evaluation Setup

Models tested: 25 MLLMs (including Claude Opus 5, Gemini 3.8 Flash, Kimi K3, GPT‑5.6‑sol, Qwen3.5‑27B, etc.).

Input: 128 uniformly sampled video frames with timestamps.

Protocol: Questions fed sequentially; each model’s own previous answers remain in context.

Metrics: Per‑turn accuracy, episodic dependency accuracy, answer dependency accuracy. Overall score = average of the three. Answer dependency uses strict chain scoring: a turn counts only if it and all its prerequisite historical answers are correct.

Results

Top models by overall score (percentage):

Claude Opus 5 — 45.7

Gemini 3.8 Flash — 42.4

Kimi K3 — 42.2

GPT‑5.6‑sol — 41.7

Qwen3.5‑27B — 34.3

Figure 5: Top‑11 models by overall score
Figure 5: Top‑11 models by overall score

Even the best model, Claude Opus 5, achieves only 35.7% on answer dependency, indicating that chaining multiple correct answers remains a major challenge.

Analysis: Dependency Distance, Fan‑In, and Error Propagation

Distance effect: Averaged across all models, accuracy on dependent questions drops from 30.2% at distance 1 to 23.1% at distance ≥5.

Multiple prerequisites: Questions that require several prior turns (higher fan‑in) are harder.

Error propagation: Replacing a model’s own prior answers with ground‑truth answers substantially improves subsequent answer‑dependency performance, confirming that early mistakes cascade.

Figure 6: Impact of dependency distance, parent count, dialogue position, and history correctness
Figure 6: Impact of dependency distance, parent count, dialogue position, and history correctness

Resources

Project page: https://videomembench.github.io/

Paper (arXiv): https://arxiv.org/abs/2610.40131

Code (GitHub): https://github.com/Oliiveralien/VideoMemBench

Dataset (Hugging Face): https://huggingface.co/datasets/VideoMemBench/data

The benchmark aims to push models toward more coherent, reliable multi‑turn video conversations by making memory of established information a first‑class evaluation criterion.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

QwenGeminivideo understandingGPTmultimodal modelsMLLMKimimemory benchmarkClaude Opus 5answer dependencyepisodic dependencyVideoMemBench
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.