Apple's IVT Internalizes Visual Reasoning for 5x Faster Proactive Video Prediction

Apple's research team introduces Internalized Visual Thinking (IVT), a post-training framework for Proactive Video Reasoning that addresses the latency bottleneck of Visual Chain-of-Thought (Visual CoT). The work targets two tasks: Early-event Prediction (identifying an action before it ends) and Next-event Prediction (forecasting the next action before it occurs), both of which require modeling future visual dynamics but demand real-time responsiveness.

Visual CoT Is Effective but Costly

The paper first compares Answer-Only supervised fine-tuning (SFT) with Visual CoT. On Early-event Prediction, Visual CoT improves accuracy but raises average latency from 1.07 seconds to 6.56 seconds. On Next-event Prediction, latency similarly jumps from 1.32 seconds to 6.55 seconds, while accuracy gains are inconsistent. Explicitly generating future frames is expensive and does not reliably translate to better results.

To isolate the cause, the authors run an Oracle experiment: instead of model-generated future frames, they feed ground-truth future frames. With real future frames, ROUGE-L on Early-event Prediction rises from 34.4 (Answer-Only SFT) to 41.9, compared to 36.4 with generated frames. A similar trend appears for Next-event Prediction. This confirms that future visual information itself is valuable; the bottleneck lies in the cost and error propagation of explicit pixel-level generation.

IVT: Train with Future Prediction, Infer Without It

IVT's core idea: during training, the model predicts both the text answer and the latent embedding of future video frames (Next-Embedding Prediction); at inference, the future-prediction branch is entirely removed, and the model answers directly from the observed video. The future representation is not raw pixels but a latent embedding — specifically, the paper finds Flux-VAE embeddings work best among Flux-VAE, DINOv2, and SigLIP2. Flux-VAE retains fine-grained appearance, spatial structure, and local state information critical for short-term action evolution.

Two key design findings emerge:

Joint training is essential. A two-stage approach (first train future embedding prediction, then text SFT) underperforms end-to-end joint optimization. The benefit is not merely a better video representation; the future-prediction signal continuously shapes the internal reasoning representation used for answering.

Data mixing ratio matters. Configurations favoring language supervision converge faster early but saturate sooner. Maintaining a substantial future-prediction weight throughout training yields better final results, indicating the auxiliary loss must persistently influence the model's internal representations.

Prediction horizon should be short. Short-term future is tightly coupled to current visual evidence, providing reliable supervision. As horizon grows, uncertainty rises sharply, and binding noisy long-range predictions to the language model can degrade performance. IVT is therefore suited to near-future prediction, not long-horizon planning.

Experimental Results: 6/6 Settings Beat SFT, 4/6 Beat Visual CoT, 5x Speedup

Experiments run on Ego-Exo4D, Ego4D, and EPIC-KITCHENS-100, each evaluated on Early-event and Next-event Prediction (six dataset-task settings). IVT outperforms Answer-Only SFT in all six settings. Against Visual CoT, IVT wins in four settings and sweeps all three Next-event Prediction benchmarks, showing it is not merely trading accuracy for speed.

On a unified NVIDIA B200, batch size 1, bfloat16 setup, average per-sample latency is 1.20 seconds for Answer-Only SFT, 1.22 seconds for IVT (nearly identical), and 6.56 seconds for Visual CoT. IVT delivers over 5x inference acceleration versus Visual CoT without increasing deployment cost.

Limitations and Future Directions

The authors acknowledge clear boundaries: IVT shows no clear gain on longer-horizon hop=3 anticipation tasks, and preliminary tests with V-JEPA 2.1 features as prediction targets did not yield stable improvements. Longer futures involve multiple plausible branches dependent on goals, long-term context, and procedural knowledge, not just short-term visual dynamics. Evaluation also becomes harder because a single incomplete video may correspond to multiple valid next events, and single-reference text-similarity metrics cannot fully capture predictive quality.

IVT opens a design space between Visual CoT and pure text reasoning: models can learn "how the world will change" during training and internalize that capability, while inference requires no explicit visual generation. Future work may explore hybrid schemes — e.g., lightweight latent rollout only on difficult samples — to balance test-time compute and efficiency. For real-time multimodal systems, the "think during training, answer directly at inference" paradigm offers a more efficient path.

Paper: https://arxiv.org/abs/2608.15869 (arXiv:2608.15869). Technical blog:

https://zgzxy001.github.io/blog/internalized-visual-thinking.html

.