UniWorld-View Tops WorldScore, Letting AI Infer Unseen Spaces and Create New View Videos

AI‑generated images and videos are now commonplace, but the real challenge lies in enabling AI to understand the underlying 3‑D world rather than merely producing content.

Problem Definition

Novel view synthesis requires an AI system to infer unseen parts of a scene from limited viewpoints and render them from new camera positions. The difficulty escalates with large baselines, such as predicting a side or back view from a single frontal image, because the model must preserve spatial structure, occlusion relationships, and object placement.

Limitations of Prior Approaches

Traditional reconstruction pipelines like NeRF or 3DGS need dense multi‑view data; when fed with sparse monocular video they produce holes, artifacts, or collapse entirely. Recent diffusion‑based methods treat camera pose as a simple conditioning vector, lacking explicit 3‑D modeling, which leads to severe distortion under large‑angle view changes.

Recent works (e.g., ViewCrafter, NVIDIA GEN3C) mitigate this by first building a geometric point cloud and then feeding the rendered point‑cloud image to a diffusion model, but they still suffer from two common failure modes.

Why Point‑Cloud Rendering Fails

Foreground‑background tearing: At depth discontinuities the point cloud provides no connectivity, causing foreground textures to be pulled onto the background when the view changes.

Back‑face leakage: Points have no notion of surface orientation, so points facing away from the camera become visible through the object when the camera moves behind it.

UniWorld‑View’s Solution

UniWorld‑View proposes an occlusion‑aware point‑cloud rendering pipeline consisting of two steps.

1. Double‑Reprojection (Tear‑Fix)

The source image is projected to the target view and then re‑projected back. Pixels that cannot return to their original location are marked as occluded; these regions are accumulated into an occlusion mask and removed from the rendering, allowing the diffusion model to fill them correctly.

2. Normal Filtering (Back‑Face Fix)

Each point’s normal vector is estimated and compared with the view direction; points whose normals face away from the camera are discarded, eliminating back‑face artifacts.

These two operations clear erroneous geometric signals, leaving only reliable structure and explicit holes for the generative model.

Dual‑Branch Architecture

After geometry is regularized, UniWorld‑View injects conditioning through two parallel streams.

Geometry Flow: The point‑cloud rendering and occlusion mask are encoded and added to the latent variable, anchoring the generation to the correct 3‑D layout.

Appearance Flow: A newly designed Ref‑DiT module treats the target‑view features as queries and the source video features as keys/values in a cross‑attention mechanism, allowing the model to “look up” missing textures from the original video and repair any residual artifacts.

The combination of a geometry‑only branch and an appearance‑only branch enables consistent spatial structure and high‑fidelity visual quality even under large view changes.

4D Reconstruction Pipeline

UniWorld‑View decouples spatial and temporal dimensions to enable full 4‑D reconstruction.

Static keyframe generation: The first frame is frozen and a full 360° surround view is rendered, providing an appearance anchor for the whole scene.

Dynamic multi‑view video: For each subsequent frame, synchronized multi‑view videos are generated; the static keyframe aligns the first frame of each view, and iterative generation resolves self‑occlusion.

The resulting multi‑view video can be fed into downstream 4DGS optimization, achieving a complete pipeline from a single hand‑held video to a freely navigable 4‑D scene.

Results and Impact

UniWorld‑View topped the WorldScore leaderboard curated by Fei‑Fei Li’s team, demonstrating superior performance on novel view synthesis benchmarks. The code and model weights have been open‑sourced on GitHub (https://github.com/PKU-YuanGroup/UniWorld-View) and adapted to Ascend hardware.

The work illustrates how a unified “world‑model” that jointly learns geometry and appearance can push visual AI toward embodied intelligence and robotics, positioning world models as a key research direction for the next generation of AI systems.