Is Video Generation the ‘Next Token Prediction’ for Vision? Insights from the GenCeption Paper

Language models succeed by predicting the next token, unifying translation, summarization, QA and reasoning. In contrast, computer vision remains fragmented: segmentation relies on SAM, depth on Depth Anything, pose and normal estimation each need separate models. The visual field lacks a unified, extensible training objective.

Why Video Generation?

Recent ECCV 2026 work titled Video Generation Models are General‑Purpose Vision Learners (He Kaiming, Andrew Zisserman et al.) argues that text‑to‑video generation satisfies the three conditions needed for a universal visual pre‑training task: it processes semantic, spatial and temporal information jointly, and can be scaled with data and compute.

From Multi‑Step Generation to Single‑Step Perception

GenCeption builds on the open‑source text‑to‑video model WAN 2.1. The original model generates video through multiple denoising steps; GenCeption instead feeds the noise‑free latent video representation directly to the DiT backbone with the time step fixed at t = 0, performing only a single forward DiT pass.

Because WAN 2.1 uses a Rectified Flow objective, the authors negate the DiT‑predicted velocity to better align with target video latents.

Unified Task Encoding

Dense tasks (depth, surface normal, segmentation) are encoded as three‑channel images; six‑channel camera ray maps are spatially partitioned and rendered as RGB. Sparse tasks (2D/3D keypoints, camera pose) receive learnable tokens whose coordinates are regressed by an MLP. All tasks share the pretrained DiT backbone and are trained with a uniform L2 loss, using either VAE decoding (dense) or coordinate regression (sparse). Task distinction is provided via textual prompts and target data representations.

Dataset Construction

The authors synthesize 7,500 human‑centric video clips, each annotated with depth, normals, segmentation, keypoints and camera parameters. Open‑world expression segmentation leverages real datasets such as MeViS, RefCOCO and YouTube‑VOS.

Models ingest 81‑frame videos at 480×832 resolution. The 14B parameter version runs at ~8 FPS on a single v6e TPU block, consuming 42.8 GB of memory, indicating a gap to lightweight real‑time deployment.

Multi‑Task Performance

GenCeption matches or approaches specialist models on depth, normal, pose, foreground segmentation, expression segmentation and 3D human keypoint tasks. The 14B single‑task variant achieves 29.7 mAE on Sintel normals, 76.4 J&F on Ref‑DAVIS expression segmentation, and 71.8 MPJPE on EMDB 3D keypoints. Depth and pose results are comparable to dedicated models such as Depth Anything 3, D4RT and VGGT‑Ω.

Specialist (single‑task) models are fine‑tuned separately, while the Generalist (multi‑task) version is jointly trained. Joint training improves foreground segmentation, leaves expression segmentation unchanged, and yields mixed effects on normals, depth, pose and keypoints, with the latter degrading most noticeably.

Role of Textual Prompts

Beyond selecting output modalities, textual prompts enable the model to locate targets based on color, spatial relations and action descriptions, distinguishing active from passive agents and recognizing categories absent from the fine‑tuning data.

Pre‑Training Contribution

Under identical fine‑tuning data, WAN 2.1 outperforms V‑JEPA and VideoMAE V2. Across 7,500 videos (~900 k frames), average AbsRel on three depth benchmarks drops from 0.281 (V‑JEPA) and 0.154 (VideoMAE V2‑G) to 0.122 (WAN 2.1 1.3B) and 0.093 (WAN 2.1 14B). Adding more synthetic data reduces AbsRel further to 0.071 for the 14B model, using ~1.23 M fine‑tuning frames—far fewer than the ~86 M (D4RT) or ~600 M (VGGT‑Ω) frames used by the baselines.

Layer‑wise ablations show that loading more pretrained DiT layers improves convergence; loading all 40 layers yields the lowest final loss.

Generalization to Real Video

Although fine‑tuned only on synthetic human videos, GenCeption handles real‑world multi‑person scenes zero‑shot and generalizes to unseen categories such as animals, robots and stylized characters.

Limitations and Outlook

Top scores often come from specialist models; the 14B Generalist still demands substantial resources, and cross‑category ability is demonstrated mainly through qualitative examples. The paper focuses on perception tasks (depth, geometry, segmentation, pose) and does not claim full visual intelligence (reasoning, planning, interaction). Nonetheless, the results suggest that generative pre‑training endows transferable visual priors that can be converted into diverse perception capabilities, and future larger‑scale multi‑task training will be needed to confirm whether video generation can serve as the universal “next token prediction” for vision.