Why Scaling Laws Fail for Video MLLMs: Uncovering the Temporal Hacking Problem

The Scaling Law has driven impressive gains in large language model (LLM) pre‑training, and researchers have extended it to multimodal models (MLLM) with success on image tasks. However, in the video domain a puzzling “anti‑scaling law” appears: more data and larger models lead to worse performance.

From a reinforcement‑learning perspective the authors examine current video‑language modeling paradigms and discover a pervasive “temporal hacking” mechanism. Models tend to take shortcuts by attending only to a few critical frames—much like watching only the beginning and end of a movie—causing severe degradation in video understanding tasks. This shortcut learning is analogous to reward hacking, where an agent maximizes a proxy reward without achieving the true task objective.

The paper formalizes temporal hacking by mapping video‑language modeling to a Markov Decision Process (MDP). The state space consists of video frames, the action space of text tokens, and the reward function is the cross‑entropy loss on predicted tokens. When the agent optimizes this proxy reward using only a subset of frames, the resulting policy diverges from the true objective, creating a reward gap that grows with video length, explaining the observed anti‑scaling behavior.

To quantify this gap the authors define Temporal Perplexity (TPL) – the difference between the full‑video cumulative reward and the reward obtained from a single randomly sampled key frame. Experiments using perplexity as a proxy show that higher TPL scores correlate with better performance, while low TPL scores predict poor results even with large data volumes.

Two design principles are proposed to reduce temporal hacking: (1) high frame‑information density—each frame should have a distinct textual description, and (2) high inter‑frame dynamics—descriptions must reflect temporal changes. Implementing these, the authors introduce the Unhackable Temporal Rewarding (UTR) pipeline: expert models (e.g., GRiT, Grounding‑DINO) extract unique spatio‑temporal attributes per frame, a tracking algorithm (ByteTrack) builds attribute trajectories, and a bidirectional query task forces the model to reason over arbitrary temporal and spatial queries, encouraging full‑video comprehension.

Using the UTR pipeline, a new video‑language dataset (UTR‑Data) is constructed and used to fine‑tune LLaVA‑NeXT‑Video, yielding the Video‑UTR model. Benchmarks on video and image understanding tasks demonstrate state‑of‑the‑art performance, and ablation studies confirm the effectiveness of the two principles. Additional analyses show a strong positive correlation between TPL scores and video‑text dataset quality, suggesting TPL as a reliable metric for data selection.

Overall, the work reveals that improper reward design leads to temporal hacking in video MLLMs, proposes a principled metric (TPL) to diagnose the issue, and offers a concrete UTR solution that substantially improves video‑language modeling.