Open VideoChat3: A Full‑Stack 4B Video Understanding Model for Short, Long, and Streaming Videos

Motivation

Video is a primary data form for describing the physical world, and video understanding models are essential for AI to interact with that world. Existing models struggle with fine‑grained actions, hour‑long sequences, and continuous streams, requiring not only “what happened” but also “where the evidence is” and “when to respond”.

Model Overview

VideoChat3 is a 4‑billion‑parameter multimodal large language model released by research teams from Nanjing University, Shanghai AI Lab, Nanyang Technological University and Peking University. It targets “comprehensive, efficient, open‑source” video understanding.

The core visual encoder, Inflated 3D Vision Transformer (I3D‑ViT), moves temporal modeling to the vision stage. Frames are grouped into short clips; intra‑clip spatio‑temporal attention is applied, followed by temporal pooling. This yields about 16× spatio‑temporal compression without simply dropping frames.

Adaptive Frame‑Resolution Mechanism

To avoid processing every moment at full resolution, VideoChat3 introduces three stream states:

</Silence>: no evidence related to the query, model stays silent with low‑cost observation.

</Standby>: potential clues appear, model raises the resolution for the next window.

</Response>: sufficient evidence gathered, model answers and returns to low‑cost monitoring.

Ablation on the OVO‑Timing benchmark shows the adaptive policy achieves an average F1 of 35.5, higher than fixed‑low (33.5) and fixed‑high (30.5) resolutions, while using only 30.2 % of the visual budget of a constantly high‑resolution setting.

Data Collections

Three new open‑source datasets support short‑video, long‑video, and streaming scenarios.

VideoChat3‑Academic2M rewrites ~2.27 M academic Q&A pairs into evidence‑rich natural‑language answers, adding temporal cues and consistency checks.

VideoChat3‑LV116K provides ~116 k long‑video clips. After filtering low‑quality videos, the team segments them using shot boundaries, generates and verifies segment‑level descriptions, and assembles timestamped evidence for tasks such as temporal localization and summarization.

VideoChat3‑OL617K contains 617 k online streaming samples. Each sample is transformed into a causal sequence of windows and state tokens ( </Silence> , </Standby> , </Response> ) that teach the model when to answer. Adding this set raises OVO‑Timing F1 from 4.0 to 35.5 while keeping other offline metrics stable.

Four‑Stage Training Pipeline

Vision encoder pre‑training: MoonViT‑initialized I3D‑ViT learns video representations compatible with the language model.

Video‑language alignment: compact video tokens are aligned with language space.

Instruction fine‑tuning: multi‑modal description and QA data endow general instruction following.

Long‑video & streaming instruction fine‑tuning: focuses on long‑range retrieval, temporal grounding, evidence aggregation, state judgment and proactive response.

The four stages use only 25 M training samples yet achieve competitive results across many video tasks.

Benchmark Results

On offline video understanding, VideoChat3‑4B surpasses the open‑weight Qwen3‑VL‑4B on 18 of 19 metrics. Notably, it scores 70.1 on Video‑MME and leads on all TimeLens sub‑benchmarks, outperforming closed‑source GPT‑5.

For streaming video, VideoChat3 attains the best scores on four of six perception‑memory metrics (ODVBench 72.3, StreamingBench 83.0, River 42.8) and reaches an OVO‑Timing F1 of 35.5, exceeding the specialized Em‑Garde model.

Efficiency tests on NVIDIA H200 with HuggingFace Pipeline and FlashAttention‑2 show that for 512 frames the total latency is 3.596 s (vs 3.838 s for Qwen3‑VL); for 1024 frames 8.099 s vs 12.251 s; for 2048 frames latency drops from 44.449 s to 20.412 s, FLOPs from 1.515×10¹⁶ to 5.738×10¹⁵, and memory from 106.9 GB to 80.8 GB.

Conclusion

VideoChat3 delivers a comprehensive, efficient, and fully open‑source solution for video understanding, unifying offline general‑purpose ability with online proactive response. While it balances capability and cost well, further work is needed on complex temporal reasoning, response precision, and real‑world deployment.