VideoMemBench Reveals Claude Opus 5 Leads 25 Models in Video Conversation Memory
Researchers from UTS, CUHK, HKUST, and Zhejiang University introduce VideoMemBench, a benchmark testing 25 multimodal models on 326 videos with 2,476 QA rounds to evaluate episodic and answer dependency in video conversations, finding Claude Opus 5 achieves the highest composite score of 45.7 while all models struggle with long-range dependencies.
Imagine watching a basketball game and continuously asking a large model about the action. Early on you ask, "Who just scored on that drive?" The model names the player. After several more exchanges about other plays, you follow up: "How many points did that first player finish with?" To answer, the model must retrieve the player's identity from its earlier response and combine it with the final statistics. If it misidentifies the player, the answer fails even if it understands the stats.
A similar need arises when discussing a movie. After talking about an opening scene and then other characters, you point to someone on screen and ask, "Did this character appear in the very first scene we discussed?" The model must recall which scene was discussed initially and verify the character's presence. Understanding only the current frame is insufficient.
Such follow-up questions are natural in real dialogue and require the model to link the current question to prior conversation — retrieving a previously identified entity in the first case, or a previously discussed scene in the second.
Researchers from the University of Technology Sydney, the Chinese University of Hong Kong, the Hong Kong University of Science and Technology, and Zhejiang University jointly propose VideoMemBench , a benchmark that evaluates how well multimodal large language models (MLLMs) remember and correctly use information established earlier in a sustained conversation about the same video.
Most existing video QA benchmarks use independent questions: the model can answer from the video and the current question alone, without needing conversation history. VideoMemBench organizes questions into connected multi-turn dialogues so that later answers depend on previously established information, focusing on two dependency types:
Episodic Dependency — the question refers back to a previously discussed scene; the model must identify the referenced segment from the dialogue history and then use visual evidence to answer.
Answer Dependency — the question continues probing an entity, object, or event the model already identified; the model must build on its own prior answers.
The former concerns earlier scenes , the latter earlier answers .
To verify that questions truly require context, the team designed a three-role pipeline — Planner, Generator, and Judger — for dependency planning, QA generation, and quality checking, followed by human expert review of all retained dialogues. The key checks: removing relevant history should make the question unanswerable; the current question must not restate the scene or answer that needs to be retrieved; and the reference answer must be grounded in the video frames.
The final dataset contains 326 videos and 2,476 QA rounds , of which 1,387 rounds require context. It includes multi-hop references and questions that depend on multiple historical turns simultaneously.
The team evaluated 25 models . Input consisted of 128 timestamped sampled frames per video. Questions were fed sequentially in dialogue order, with each model's previous answers retained in context. Results (composite score, average of per-round accuracy, episodic dependency, and answer dependency) for the top models:
Claude Opus 5 — 45.7
Gemini 3.8 Flash — 42.4
Kimi K3 — 42.2
GPT-5.6-sol — 41.7
Qwen3.5-27B — 34.3
The composite score averages three metrics: per-round accuracy, episodic dependency accuracy, and answer dependency accuracy. Answer dependency uses stricter chain scoring : a current answer counts as correct only if it is correct and all historical answers it depends on are also correct. Thus this metric measures the reliability of the entire answer chain. Even the best model, Claude Opus 5, reaches only 35.7% on answer dependency.
Further analysis shows that as the dependency distance grows, or as the number of historical turns that must be jointly referenced increases, models find it harder to answer correctly. Averaged across all tested models, when dependency distance increases from 1 turn to at least 5 turns, accuracy on dependent questions drops from 30.2% to 23.1% .
An ablation replacing the model's own previous answers with ground-truth answers substantially improves performance on subsequent answer-dependent questions. This indicates that early answer errors can cascade and degrade later reasoning, while accurate history supports continued dialogue.
VideoMemBench aims to push models to more fully exploit information already established in video conversations, making sustained questioning and cross-scene discussion more coherent and reliable. For video assistants, understanding the current frame is not enough; the ability to pick up on what was discussed earlier is equally critical and deserves rigorous evaluation.
Resources
Project: https://videomembench.github.io/
Paper (arXiv): https://arxiv.org/abs/2610.40131
Code (GitHub): https://github.com/Oliiveralien/VideoMemBench
Dataset (Hugging Face): https://huggingface.co/datasets/VideoMemBench/data
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
