VGGRPO: 4D Latent Rewards for World‑Consistent Video Generation (ECCV 2026)

Large‑scale video diffusion models achieve impressive visual quality but often suffer from geometric drift, unstable camera trajectories, and incoherent scene structure, which limits their use in embodied AI and world‑action models.

Previous attempts either modify the generator architecture with geometry‑aware modules—risking loss of the pretrained model’s generalization—or apply reinforcement‑learning‑based post‑training that relies on RGB‑space rewards, requiring costly VAE decoding and failing on highly dynamic scenes.

VGGRPO (Visual Geometry GRPO), presented at ECCV 2026 by researchers from Google, the University of Copenhagen, and the University of Oxford, addresses this by introducing a 4‑dimensional latent‑space geometry reward for efficient, geometry‑aware video post‑training without sacrificing pretrained generalization.

The framework consists of two tightly coupled components: a Latent Geometry Model (LGM) that stitches the diffusion model’s latent variables to a geometry backbone (Any4D) via a lightweight connector, enabling direct prediction of 4D scene geometry from latent space; and a GRPO training loop that optimizes two rewards in latent space.

Camera Motion Smoothness Reward penalizes acceleration of the predicted camera pose, encouraging near‑constant‑velocity, physically plausible trajectories; smoother motion yields a reward close to 1, while jitter reduces it.

Geometry Reprojection Consistency Reward reprojects the predicted 3D structure (point cloud, depth, camera parameters, optical flow) into multiple views and compares depth maps, filtering dynamic regions with scene flow to aggregate static points, thus ensuring cross‑view 3D coherence.

Extensive experiments on static and dynamic benchmarks show that VGGRPO outperforms baselines on geometry‑related metrics and overall video quality, confirming that geometry‑aware post‑training does not compromise the pretrained model’s generative ability.

Ablation studies (Fig 4) reveal that the motion‑only reward stabilizes camera paths but leaves geometric artifacts, whereas combining it with the reprojection reward eliminates both drift and artifacts, demonstrating the complementary nature of the two components.

VGGRPO therefore provides a practical, architecture‑free approach to generate videos that are both visually realistic and physically consistent, benefiting downstream tasks such as world‑action modeling and embodied intelligence.