VideoChat3: Open-Source Full-Stack Video Understanding Model Linking Perception, Understanding, and Interaction

Video is a fundamental data modality for describing the physical world, and effective video understanding requires not only recognizing short actions but also handling hour‑long sequences and real‑time streams. The authors identify the key challenges of fine‑grained motion detection, long‑range information processing, and timely response.

Model Architecture

The paper introduces the Inflated 3D Vision Transformer (I3D‑ViT), which moves spatiotemporal modeling to the visual encoder. Consecutive frames are grouped into short clips; within each clip a joint spatiotemporal attention is applied, followed by temporal pooling. This design achieves roughly 16× compression of visual tokens without simply dropping frames, preserving dense evidence for the language model.

Adaptive Resolution Mechanism

To avoid constant high‑resolution processing, VideoChat3 defines three streaming states: </Silence>: No evidence related to the query; the model stays silent with low‑cost observation. </Standby>: Potential evidence detected; the model raises the resolution for the next window. </Response>: Sufficient evidence gathered; the model answers and returns to low‑cost monitoring.

This closed‑loop allows visual budget to be allocated dynamically, focusing computation on informative segments. Ablation on the OVO‑Timing benchmark shows an average F1 of 35.5, surpassing fixed‑low (33.5) and fixed‑high (30.5) resolutions while using only 30.2% of the visual budget of a constantly high‑resolution setup.

Data Resources

Three new open datasets are released:

VideoChat3‑Academic2M : ~2.27 M academic video Q&A pairs rewritten to include observable evidence and temporal cues.

VideoChat3‑LV116K : ~116 K long‑video clips with timestamped evidence, enabling fine‑grained temporal localization and summarization.

VideoChat3‑OL617K : ~617 K online streaming samples where each window is labeled with the three states above, turning offline QA into a proactive “when‑to‑answer” task. Adding this set raises OVO‑Timing F1 from 4.0 to 35.5 while keeping other offline metrics stable.

Four‑Stage Training Pipeline

Visual encoder pre‑training: Initialize I3D‑ViT from the image‑pre‑trained MoonViT.

Video‑language alignment: Learn compact video token ↔ language space mapping.

Instruction fine‑tuning: Train on diverse image/video description and QA data for general instruction following.

Long‑video & streaming instruction fine‑tuning: Emphasize long‑range retrieval, temporal grounding, evidence aggregation, state judgment, and proactive response.

Only 25 M training samples are used across the four stages, yet the model attains strong performance on multiple benchmarks.

Evaluation Results

On offline video understanding, VideoChat3‑4B achieves competitive scores in temporal perception, long‑video comprehension, complex reasoning, and temporal grounding. Compared with the same‑size open model Qwen3‑VL‑4B, it improves 18 out of 19 directly comparable metrics, including a 70.1 score on Video‑MME and leading results on TimeLens.

For streaming video, VideoChat3 tops four of six perception‑memory metrics (ODVBench 72.3, StreamingBench 83.0, River 42.8) and attains the highest average F1 of 35.5 on OVO‑Timing, outperforming the specialized Em‑Garde model.

Efficiency Analysis

Because the visual encoder performs early spatiotemporal compression, the model reduces the load on the language model, which scales poorly with long sequences. Under NVIDIA H200, Hugging Face Pipeline, and FlashAttention‑2, the following latencies were observed:

512 frames: 3.596 s (VideoChat3) vs 3.838 s (Qwen3‑VL).

1024 frames: 8.099 s vs 12.251 s.

2048 frames: latency drops from 44.449 s to 20.412 s, FLOPs from 1.515×10¹⁶ to 5.738×10¹⁵, and memory from 106.9 GB to 80.8 GB.

These results confirm that the “more work on the vision side, less on the language side” trade‑off becomes increasingly beneficial as video length grows.

Conclusion

VideoChat3 delivers a comprehensive, efficient, and fully open‑source solution for general video understanding. It bridges perception, understanding, and interaction through a native spatiotemporal encoder, adaptive resolution budgeting, and a carefully staged training regime. While it sets a strong baseline, future work can further improve complex temporal reasoning, proactive response precision, and real‑world deployment efficiency.