From LLMs to World Models: The Next AI Revolution

Breaking the Wall: The Core Pain of Current AI – Description Without Experience

GPT‑4 sparked optimism about AGI by passing exams and writing code, yet it fails a simple physical reasoning test: when a cube slides on a smooth table, the model inconsistently answers whether friction aligns with motion, and even when wrong it can fabricate a plausible causal explanation.

Key Insight : Large language models learn statistical language patterns, not the physical laws governing the world. As Yann LeCun notes, the AI Moravec paradox persists because models fit language correlations without modeling reality.

LLMs predict the next word based on text, mastering linguistic regularities but lacking causal understanding—e.g., they know "fire is hot" but cannot explain why touching fire burns skin or predict the outcome of reaching for a flame.

Multimodal models add perception (images, video) but still miss dynamic experience. As Fei‑Fei Li explains, they map static pixels to words; showing a billiard table photo lets the model describe the scene, yet it cannot reliably predict ball trajectories after a strike.

Definition: World Models – From Understanding to Prediction to Action

Rooted in Kenneth Craik’s 1943 notion of a "small-scale world model," the modern definition (Dyna 1991; Google Brain "World Models" 2018) is:

World Model = Observation (V) + Prediction (M) + Internal Action Learning (C)

It abstracts reality into a latent space, predicts the next state S' after action A, and plans optimal actions—three core traits that distinguish it from LLMs and multimodal models.

Representation : Recognizes objects, positions, attributes, and spatial/causal relations—beyond static perception.

Prediction : Simulates dynamic evolution using physics (e.g., predicts a ball’s trajectory after a hit).

Planning & Control : Internally evaluates multiple action possibilities, selects the optimal one, and closes the perception‑reasoning‑action loop.

Meta product design lead Yiqi Zhao describes world models as a "miniature parallel universe" that gives AI a true worldview, enabling observation‑reasoning‑action like humans.

Why World Models Became the 2024 Focus

Three converging forces made world models a hot topic:

1. LLM Scaling Hits a Ceiling

From GPT‑3 to GPT‑4, parameter growth yielded diminishing returns while training costs exploded to hundreds of millions of dollars. Researchers estimate high‑quality text data will be exhausted by 2026‑2028. Richard Sutton warns that LLMs lack goals and evaluation criteria, so scaling alone cannot achieve AGI.

2. Embodied Intelligence Demand Explodes

2024 saw massive funding for humanoid robots (Figure AI, 1X Technologies) and the maturation of autonomous driving and industrial robots. These applications require physical understanding, causal reasoning, and action prediction—capabilities LLMs and current multimodal models cannot provide.

3. Engineering Maturity Makes Practice Feasible

Abundant video and sensor data, advanced multimodal perception, and powerful GPU/TPU clusters now support large‑scale video generation and 3D simulation. OpenAI’s Sora demonstrated that video pre‑training can endow models with physical intuition, turning world modeling from theory into a visible reality.

Global Technical Roadmaps – Divergent Paths to World Models

Overseas leaders (OpenAI, Google, Meta, Fei‑Fei Li’s World Labs) focus on two layers: world generation and agent training. Four representative routes:

Route 1 – Video Generation (Sora, Genie)

Generates physically consistent video, making training data visible and commercializable (film, advertising, games). Advantage: easy data acquisition, fast commercialization. Limitation: knowledge remains implicit in weights, hard to transfer to robots.

Route 2 – 3D Generation (Marble)

Reconstructs explicit 3D scenes from language or images, providing precise spatial structure for robotics. Advantage: facilitates physical simulation and planning. Limitation: scarce high‑quality 3D data and high computational cost.

Route 3 – Virtual‑World Agent Training (Google SIMA 2)

Places agents in game‑like environments to learn complex instructions and cross‑domain generalization. Advantage: bridges perception‑action gap, enabling autonomous decision‑making. Limitation: simulation‑reality gap persists.

Route 4 – Abstract Structure Learning (Yann LeCun’s JEPA)

Compresses reality into high‑dimensional latent representations, predicting only task‑relevant future structure. Advantage: low compute, better causal capture for robotics. Limitation: outputs are invisible, evaluation standards are still open.

Domestic Strategies – Five Tailored Routes

Chinese giants leverage strengths in data annotation, 3D modeling, game engines, and multimodal foundations to pursue five pragmatic paths:

Route 1 – Niche Domain World Models

Focus on specific tasks (desktop operations, cooking, assembly) with closed‑loop "state‑action‑result" triples. Advantage: rapid deployment, high‑quality data, complete physical loop. Limitation: limited generalization.

Route 2 – Large‑Scale 3D Data Ingestion

Combine physics engines and 3D simulators to generate massive dynamic scenes (collision, fluid, cloth). Advantage: precise physical modeling, strong generalization. Limitation: high data collection cost.

Route 3 – Expert Physical Annotation

Employ physics‑trained annotators to label causal outcomes (e.g., glass fracture patterns) that crowdsourcing cannot judge. Advantage: injects domain knowledge; Limitation: scalability.

Route 4 – Game‑Engine Training Grounds

Use mature engines from Tencent, NetEase, miHoYo to produce virtually unlimited, automatically labeled interaction data. Advantage: low cost, perfect physical consistency. Limitation: style gap between game physics and real world.

Route 5 – Incremental Multimodal Evolution

Build on existing multimodal models by adding temporal dynamics, conditional action prediction, and causal inference modules, avoiding full‑scale retraining. Advantage: fast, low‑cost, leverages existing assets. Limitation: may not overcome fundamental multimodal architecture limits.

Common Trends and Future Outlook

All routes converge on the same goal: enable AI to move from "outputting information" to "understanding, reasoning, and acting in the world." Companies are pursuing multi‑route strategies, and the eventual winner will likely fuse explicit 3D structure with dynamic video evolution and virtual‑world training.

Impact on Roles and Industries

AI Trainer – From Data Worker to World‑Rule Designer

Quantity no longer matters; quality and depth of physical knowledge become critical. Trainers must build physical intuition, causal thinking, design adversarial physical samples, and shape data ecosystems.

Algorithm Engineer – From Model Tuning to System Construction

Engineers need cross‑disciplinary expertise: deep learning, physics simulation, 3D modeling, reinforcement learning, and robot control, integrating perception, world generation, and planning into a cohesive system.

Product Manager – From Experience Design to Scenario Deployment

Product focus shifts to solving real‑world problems, balancing technical feasibility with scene requirements, and ensuring safety and reliability for physical deployments.

Industry Transformations

Robotics : World models give robots internal simulations, enabling rapid adaptation (e.g., learning to pour coffee in simulation then handling diverse real cups).

Autonomous Driving : Predictive world engines can simulate multiple traffic participant trajectories, moving from perception‑reaction to proactive planning toward Level‑5 autonomy.

Content Creation & Gaming : AI will define a world view and autonomously generate evolving narratives, reducing production costs and enabling real‑time, AI‑driven game worlds.

AI Agents : Internal world simulators let agents plan and act autonomously, transforming them from tool callers to genuine assistants.

Manufacturing : Intelligent robots will self‑adapt to new products and lines, achieving flexible, smart production.

Outlook: World Models as the Essential Path to AGI

World models are still in an early, exploratory stage—issues like error accumulation, out‑of‑distribution generalization, simulation‑to‑reality transfer, and lack of unified evaluation remain unsolved. Yet this immaturity creates vast opportunity: participants can shape standards and drive breakthroughs.

Just as the community’s understanding of Transformers in 2017 was nascent, today we recognize world models as the next pivotal architecture. They are not a fleeting hype but a necessary evolution for AI to truly comprehend and manipulate the physical world.

In summary, AI’s journey is moving from mastering language, to perceiving the world, to finally understanding and acting within it. World models represent the bridge to that future, heralding a new era of human‑AI symbiosis.