Beyond Natural Language: How PhiZero Teaches World Models a Physical Language

Natural language enables AI to read human knowledge, but PhiZero asks whether AI can also learn a “physical language” that directly represents fine‑grained changes in the real world. By focusing on state transitions, actions, and causal dynamics, the model aims to predict how the world evolves.

Key capabilities

1. Physical video generation : PhiZero can model coherent processes such as seawater striking rocks, liquid pouring into a container, or an object exploding in hot oil, ensuring each frame is realistic and the effects persist over time.

2. Motion transfer : The model extracts the underlying state change from a human motion and re‑applies it to different bodies, e.g., mapping a human movement to a Unitree G1 humanoid robot or transferring a hand‑grasp sequence to a Sharpa dexterous hand, without requiring paired training videos.

3. Simulation‑to‑reality bridging : By preserving the dynamic process from a simulated clip and rendering it onto a real‑world initial appearance, PhiZero generates consistent videos across the simulation and real domains.

Interactive world modeling

PhiZero accepts continuous inputs and control signals; when the viewing direction or motion changes, the model updates the world while maintaining consistent layout, landmarks, and spatial relationships. Driving actions in a road scene lead to distinct future videos depending on the control trajectory.

In robot tasks, trajectories such as grasping or scooping are first encoded into physical language and then rendered as detailed interaction videos.

Why realism alone is insufficient

High‑fidelity video does not guarantee physical plausibility. Correct modeling requires understanding state transitions, the impact of actions, and causal unfolding of events—capabilities that traditional pixel‑space generators lack.

Physical language definition

Through self‑supervised learning on large‑scale real videos, PhiZero learns a compact discrete token set called “Physical Language”. These tokens encode how objects move, interact, and how a scene transitions from one moment to the next, separating dynamic information from static visual details.

Method: Reason‑then‑Render

Stage 1 – Physical Language Tokenizer : Detects differences between consecutive world states and encodes them into a short symbol sequence; the first frame supplies static appearance, which a pretrained diffusion decoder can later restore.

Stage 2 – Physical Language Reasoner : Takes the current image and action intent, autoregressively generates future symbols, and the diffusion decoder renders the predicted evolution into video.

Identify the current world → Infer subsequent state changes → Render visual results

A 4‑second, 8 FPS, 512×896 video can be represented after the first frame with only 256 discrete symbols , versus 44,800 continuous visual tokens used by a standard video VAE, while still achieving superior quality.

Training data

The team filtered ~ 50 k hours of real‑world video to ~ 10 k hours of unlabeled clips for the tokenizer. Together with simulated video, they assembled ~ 5 million 4‑second segments , from which ~ 1 million high‑quality physics samples were selected to train the reasoner.

Experimental validation

For generation, PhiZero leads on the Physics‑IQ Verified , PhyGround , and WorldModelBench benchmarks, showing better adherence to collisions, gravity, and continuous reactions than pixel‑based models.

For physical understanding, it achieves competitive or leading results on IntPhys2 , LikePhys , and YoCausal , mapping video pairs into physical language space to detect implausible events.

Future outlook

Scaling up data and model size, extending prediction horizons, and integrating multimodal reasoning could broaden physical language’s expressive power, aid embodied agents in planning, and enable skill transfer across diverse robot morphologies.