Is Video Generation the Vision Field’s Next ‘Next‑Token Prediction’? A Deep Dive into GenCeption

Background

Language models succeed by predicting the next token, unifying translation, summarization, QA and reasoning. In contrast, computer‑vision tasks remain fragmented: segmentation uses SAM, depth uses Depth Anything, pose, normal and camera pose each rely on separate models.

Research Question

The vision community lacks a single, extensible training objective. The ECCV 2026 paper Video Generation Models are General‑Purpose Vision Learners by He Kaiming, Andrew Zisserman and others asks whether text‑to‑video generation can serve as the “next‑token prediction” for vision.

GenCeption Overview

GenCeption repurposes the open‑source text‑to‑video model WAN 2.1 into a universal video‑perception model. Instead of multi‑step denoising, it feeds the noise‑free latent variable of a video directly to the DiT backbone with time step t=0, performing a single forward pass.

All dense tasks (depth, surface normal, segmentation) are encoded as three‑channel images; a six‑channel ray‑map encodes camera rays. Sparse tasks (3‑D keypoints) receive a learnable token whose coordinates are predicted by an MLP. The shared DiT backbone is trained with a uniform L2 loss; dense tasks reuse a VAE decoder, while sparse tasks compute loss in output space.

Dataset and Training

The authors synthesize 7 500 human‑centric video clips, each annotated with depth, normal, segmentation, keypoints and camera parameters. Open‑world expression segmentation also incorporates real datasets such as MeViS, RefCOCO and YouTube‑VOS.

Inputs are 81‑frame videos at 480×832 resolution. The 14 B‑parameter model runs at ≈8 FPS on a v6e TPU block, consuming 42.8 GB of memory.

Single‑Task Performance

On depth‑related benchmarks the 14 B model achieves an absolute relative error (AbsRel) of 0.093, improving over WAN 2.1‑1.3 B (0.122). On Sintel surface normals the mean angular error (mAE) is 29.7. Ref‑DAVIS expression segmentation reaches 76.4 J&F, and EMDB 3‑D human keypoint estimation attains 71.8 MPJPE, comparable to specialized models such as Depth Anything 3 and D4RT.

Multi‑Task Generalist vs Specialist

Specialist models are fine‑tuned per task, while the Generalist version is jointly trained on all tasks. Joint training improves foreground segmentation, leaves expression segmentation unchanged, yields mixed results on normals, and slightly degrades depth, camera pose and 3‑D keypoint performance.

The authors attribute the degradation to explicit coordinate regression breaking the continuous pixel space learned by the generative model and to the learnable token perturbing the pre‑trained DiT attention.

Zero‑Shot Language‑Guided Perception

Using natural‑language prompts, GenCeption can locate and segment objects based on color, spatial relations and actions, distinguish active from passive agents, and recognize categories absent from the fine‑tuning data.

Impact of Generation Pre‑Training

When fine‑tuned on the same 7 500‑clip, ~90 K‑frame dataset, V‑JEPA, VideoMAE V2‑G and WAN 2.1 are compared. WAN 2.1‑1.3 B reduces depth AbsRel from 0.281 (V‑JEPA) and 0.154 (VideoMAE V2‑G) to 0.122; the 14 B version further lowers it to 0.093, and with additional synthetic data reaches 0.071, despite using far fewer post‑training frames than D4RT (86 M) or VGGT‑Ω (600 M).

Layer‑wise ablation shows that loading more pre‑trained DiT layers consistently lowers training loss, with the full 40‑layer load achieving the lowest final loss.

Discussion

These results suggest that visual priors learned during video generation can be transferred to diverse perception tasks, but the evidence does not yet confirm video generation as a universal “next‑token prediction” for vision. The 14 B model remains resource‑intensive, and many gains rely on single‑task specialists and qualitative examples.

Conclusion

GenCeption demonstrates that generative pre‑training provides a strong foundation for multi‑task vision, yet larger‑scale multi‑task training is required to validate whether a single set of weights can truly replace task‑specific models.