Why Video World Models Stay in Pixel Grids and How SCoPE’s Ray‑Space Encoding Frees Them

Problem with Existing Video DiT Coordinates

Current video diffusion transformers (DiT) organize tokens on a (u, v, t) tensor grid that records frame, row, and column positions but lacks any representation of the camera’s geometry relative to the scene. When the camera moves, the same object receives different grid addresses, causing ambiguous mappings between grid locations and world positions.

SCoPE: Ray‑Space Positional Encoding

SCoPE (Sightline‑Coordinate Positional Encoding) computes a Plücker ray for each video token once the camera trajectory is known. Each ray is projected, normalized, and passed through a scale‑gate (Normalize‑Gate‑Inject) before being written into the attention query and key vectors. This adds less than 0.1 % of the original model’s parameters and lets attention compare both visual content and geometric ray intersection.

Interactive Camera Control Demo

Using only a single static image and a sequence of WASD commands, the model can move forward, backward, and laterally, generating coherent video without any game engine, manual 3‑D reconstruction, or additional modules. The ray‑space encoding improves both camera control fidelity and cross‑view/revisit consistency.

Ablation Study

Removing the content‑ray cross term raises rotation error to 0.135, while removing the pure‑ray term raises it to 0.159, confirming that both components are essential for performance.

Scaling Benefits

When the backbone size grows from 5 B to 14 B parameters, SCoPE’s advantage in translation error expands from 12 % to 25 %, and its improvement in Fréchet Video Distance (FVD) grows from 23 % to 47 %. Larger models can exploit the geometric information more effectively.

Revisit Consistency

In a closed‑loop trajectory test, the LPIPS recovery metric improves from 0.102 (best prior method) to 0.587, demonstrating that the same observation point retains the same ray address, enabling the model to reconnect distant frames without extra memory modules.

Generalization to Stylized Inputs

Six hand‑drawn concept images, each paired with a target camera trajectory, show that ray coordinates remain valid regardless of visual style, allowing the model to transfer learned geometric relationships to new content distributions.

Conclusion

SCoPE shifts the foundational coordinate system of video world models from a pixel‑grid to a ray‑space representation. By embedding camera rays directly into attention, tokens are organized by their observation of the world, supporting robust camera control, cross‑view consistency, and future interactive, navigable generation.