VideoMemBench: Claude Opus 5 Leads 25 Models in Video Conversation Memory Test
VideoMemBench evaluates how well multimodal large language models recall and use prior dialogue context in video conversations, testing 25 models on episodic and answer dependencies; Claude Opus 5 scores 45.7 overall but answer dependency remains only 35.7%, revealing struggles with long-range dependencies and error propagation.
VideoMemBench is a new benchmark designed to assess whether multimodal large language models (MLLMs) can remember and correctly use information established earlier in a video-grounded conversation. The benchmark was introduced by researchers from the University of Technology Sydney, the Chinese University of Hong Kong, the Hong Kong University of Science and Technology, and Zhejiang University.
Motivation: Natural Follow‑Up Questions Require Memory
The article illustrates the need with two everyday scenarios:
Basketball game: After identifying a player who scored, a later question asks for that player’s total points. The model must retrieve the player’s name from its own earlier answer and combine it with game statistics.
Movie discussion: After talking about an opening scene, a later question asks whether a character currently on screen appeared in that initial scene. The model must recall which scene was discussed first and check the character’s presence.
Both examples require linking the current question to prior dialogue — either to a previously mentioned scene (episodic dependency) or to a previously given answer (answer dependency).
Benchmark Design: Two Dependency Types
VideoMemBench organizes questions into multi‑turn dialogues around the same video, explicitly creating two kinds of dependencies:
Episodic Dependency — the question refers back to a specific scene discussed earlier; the model must identify that scene from the conversation history and then answer using the video.
Answer Dependency — the question builds on an entity, object, or event the model itself identified in a previous turn; the model must chain its own prior answers to answer correctly.
Data Construction: Planner–Generator–Judger Pipeline with Human Verification
To ensure each dependent question truly requires history, the team used three automated roles:
Planner — designs the dependency structure (which prior turns the new question relies on).
Generator — writes the actual question and reference answer.
Judger — checks that removing the relevant history makes the question unanswerable, that the question does not restate the needed prior information, and that the reference answer is grounded in the video.
Human experts then reviewed every retained dialogue. The final dataset contains 326 videos and 2,476 QA turns , of which 1,387 turns depend on prior context. Dependencies span multiple turns and some questions require multiple historical links simultaneously.
Evaluation Setup
Models tested: 25 MLLMs (including Claude Opus 5, Gemini 3.8 Flash, Kimi K3, GPT‑5.6‑sol, Qwen3.5‑27B, etc.).
Input: 128 uniformly sampled video frames with timestamps.
Protocol: Questions fed sequentially; each model’s own previous answers remain in context.
Metrics: Per‑turn accuracy, episodic dependency accuracy, answer dependency accuracy. Overall score = average of the three. Answer dependency uses strict chain scoring: a turn counts only if it and all its prerequisite historical answers are correct.
Results
Top models by overall score (percentage):
Claude Opus 5 — 45.7
Gemini 3.8 Flash — 42.4
Kimi K3 — 42.2
GPT‑5.6‑sol — 41.7
Qwen3.5‑27B — 34.3
Even the best model, Claude Opus 5, achieves only 35.7% on answer dependency, indicating that chaining multiple correct answers remains a major challenge.
Analysis: Dependency Distance, Fan‑In, and Error Propagation
Distance effect: Averaged across all models, accuracy on dependent questions drops from 30.2% at distance 1 to 23.1% at distance ≥5.
Multiple prerequisites: Questions that require several prior turns (higher fan‑in) are harder.
Error propagation: Replacing a model’s own prior answers with ground‑truth answers substantially improves subsequent answer‑dependency performance, confirming that early mistakes cascade.
Resources
Project page: https://videomembench.github.io/
Paper (arXiv): https://arxiv.org/abs/2610.40131
Code (GitHub): https://github.com/Oliiveralien/VideoMemBench
Dataset (Hugging Face): https://huggingface.co/datasets/VideoMemBench/data
The benchmark aims to push models toward more coherent, reliable multi‑turn video conversations by making memory of established information a first‑class evaluation criterion.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
