World Models for Generalist Agents: From Sequential Decisions to Actionable Predictions

The article provides a conceptual introduction to world models for generalist agents based on the paper An Introduction to World Models for Generalist Agents by Zhihui Zhu (Ohio State University, arXiv:2610.30789). It organizes the discussion around sequential decision-making, contrasting prediction with action, and surveys how different methods learn and use environment models.

1. Introduction

Traditional agents train in fixed environments and tasks. Generalist agents must transfer experience across varying environments and goals. The paper adopts a broad definition of a world model: a learned representation that captures aspects of environment structure, state, or dynamics and serves understanding, prediction, or decision-making. A world model need not replicate the entire world or reconstruct every pixel; dynamics models predicting state transitions are one important type but not the only form.

Policy answers "what action to choose given current information and task"; world model answers "what are certain aspects of the environment and how do they evolve." They are distinct capabilities but can share representations, training signals, or network architectures. An effective policy may implicitly exploit environment knowledge, yet that does not prove it contains an explicit, queryable, reliable dynamics model.

Figure 1: Unified view of world modeling and policy learning.

2. Generalist Agents: A Sequential Decision-Making Perspective

2.1 General Formulation: Partially Observable Markov Decision Process

The paper uses a POMDP to describe the agent-environment loop. Hidden state sₜ, action aₜ, observation oₜ; environment e determines transition and observation mechanisms, task g defines the goal.

State transition: p(sₜ₊₁ | sₜ, aₜ, e) Observation generation: p(oₜ | sₜ, e) Interaction history: hₜ = (o₀, a₀, …, aₜ₋₁, oₜ) Policy: π(aₜ | hₜ, g) Three common confusions: (1) camera images ≠ full state (occlusions, hidden velocities require history); (2) conditioning on environment e does not mean the agent knows environment identity or parameters; (3) even with visible physical state, unknown dynamics make past transitions useful for environment identification.

Figure 2: Partially observable interaction loop.

The learning objective maximizes expected cumulative reward over distributions of environments and tasks, not just a single fixed task. The paper assumes environment and task are fixed within an episode, treats task as given, initially fixes agent embodiment, and takes reward as a given signal (not implying reward design is solved).

2.2 Control, Estimation, and Learning in Sequential Decisions

Whether dynamics are known and whether state is fully observable determine which parts require computation, inference, or learning. The paper uses this to situate modern world models within classical decision theory.

2.2.1 Known Dynamics, Fully Observable State: Control and Dynamic Programming

If state is visible and transition mechanism known, the core problem is action selection. Dynamic programming links current and future rewards via value functions and Bellman recursion; under linear dynamics and quadratic cost, the Linear Quadratic Regulator (LQR) yields a feedback control law via Riccati recursion. Model Predictive Control (MPC) optimizes a finite action sequence from the current state, executes only the first action, then re-plans with the new state , maintaining closed-loop control without committing to a full open-loop plan. This shows that having an environment model is only a starting point; the model must combine with task goals and optimization to produce useful actions — prediction alone does not automatically yield decisions.

2.2.2 Known Dynamics, Partially Observable State: Filtering and State Estimation

When the model is known but state is not directly observable, the agent maintains a belief over state p(sₜ | hₜ). Update proceeds in two steps: predict new state distribution from old belief, action, and transition model; then correct with new observation via Bayes. In linear Gaussian systems, the Kalman filter efficiently represents uncertainty with mean and covariance; more general systems may use particle filters. Under linear-quadratic-Gaussian assumptions, state estimation and optimal control can be separated. However, this separation is not universal for all POMDPs; in nonlinear or multi-modal uncertainty, replacing full belief with a single state estimate may lose decision-critical information. For world models, latent representations must not only "remember the past" but also preserve uncertainty that affects decisions.

2.2.3 Unknown Dynamics: System Identification and Reinforcement Learning

When transition mechanism is unknown, system identification estimates environment models from trajectories (e.g., least-squares for fully observable linear systems, state-space model estimation for POMDPs, neural networks for complex nonlinear dynamics). Reinforcement learning organizes learning directly around cumulative return. Model-based RL explicitly learns an environment model for planning or generating training experience; model-free RL learns policy or value without requiring an explicit transition distribution. A model-free policy may encode environmental regularities, but that is not the same as having a callable predictive model.

For generalist agents, difficulties also stem from data acquisition: real interaction is slow, expensive, potentially unsafe, hard to reset; sim-to-real gaps exist; large image/video datasets lack action labels and rewards. The problem becomes not just "which RL algorithm" but how to leverage heterogeneous experience to learn transferable world knowledge.

3. Learning to Act: Policies and World Models

This chapter categorizes methods by prediction target and conditioning information. A single interaction trajectory with actions provides supervision for both "history+task → action" and "history+action → next observation"; action-less video still provides temporal prediction signals. The same data can support different objectives, and different objectives cannot be distinguished merely by network architecture names.

Figure 3: Policies, world models, and latent representations.

3.1 Learning Policy Directly from Demonstrations

Behavior cloning maximizes likelihood of demonstrated actions given observations, history, and task — essentially learning the conditional distribution of behavior in the data. If demonstrations come from multiple operators or policies, the model may learn a mixture, not necessarily optimal actions. This perspective accommodates modern variants: action tokenization or chunking with autoregressive prediction; Decision Transformer conditions on desired return to learn actions from offline sequences; RT-1, RT-2, OpenVLA fit within conditional policy learning; Diffusion Policy models action distribution via denoising but training remains demonstration-supervised. The paper emphasizes distinguishing model form from learning objective: using a Transformer ≠ reinforcement learning; using a diffusion model ≠ having learned environment dynamics. The key is what the model is asked to predict and what the loss function rewards.

Limitations of direct policy learning: demonstration coverage and distributional shift. Small execution errors push the system into rarely seen states where errors compound; imitation objective does not guarantee systematic improvement over demonstration quality. Pretrained vision-language models provide semantic knowledge but cannot automatically eliminate these issues. Without an explicit action-consequence model, policies struggle to compare unexecuted candidate actions.

3.2 Learning World Models for Planning and Control

The core of world model learning is extracting regularities from experience that can serve future prediction and decision-making. The paper further distinguishes observation-space models, latent-space models, and how models translate into policies.

3.2.1 Observation-Space World Models

Action-conditioned models learn pθ(oₜ₊₁ | hₜ, aₜ), answering "if I take this action now, what might I see next?" In partially observable environments, full history or an effective summary is crucial because the same image may correspond to different velocities, contact states, or hidden conditions. Training can use probabilistic likelihood, diffusion generation, or deterministic regression. Models without current-action conditioning learn pθ(oₜ₊₁ | hₜ) or, in passive video, pθ(oₜ₊₁ | o₀:ₜ). "No action conditioning" primarily means the current candidate action is not input ; past actions may still be in history. The distinction is substantive: passive models learn object changes, scene structure, and common motions but often mix consequences of different behaviors present in the data. They predict "what usually happens," not necessarily "what happens if I choose this action." Thus large-scale video pretraining is a valuable prior source, but to serve action it typically still requires action grounding and interactive validation.

3.2.2 Latent-Space World Models

Latent models first map history to a representation zₜ = φ(hₜ), then predict latent state evolution, e.g., pθ(zₜ₊₁ | zₜ, aₜ). The encoder may process only the current observation or use recurrent updates to integrate history, depending on what information must be retained. Latent representations are not equal to true physical states and need not correspond one-to-one. Their advantage is reducing pixel-level modeling burden, learning decision-relevant changes in a more compact space. However, "low dimension" and "easy to predict" do not equal "suitable for control."

A typical risk is representation collapse : if all inputs encode to the same constant, next latent state is trivially predictable but useless for decisions. Hence low prediction error alone cannot prove representation effectiveness. The paper discusses several constraint paths: PlaNet and Dreamer preserve information via observation reconstruction; joint embedding predictive architectures construct prediction constraints in representation space; MuZero emphasizes reward, value, and policy-relevant information. The ideal criterion is control sufficiency : using only the latent representation, one can achieve the optimal return attainable with full history. This is a benchmark for representation quality, not a claim that current methods universally satisfy it. Regularization against collapse is a necessary consideration but cannot substitute for testing decision-making capability.

3.2.3 From Prediction Models to Policy

Three pathways:

Online planning at inference time. Agent evaluates candidate action sequences in the model, selecting by distance to goal state or predicted return. MPC executes only the first action, then re-plans with real feedback; tree search also uses predictive models. Re-observation mitigates error accumulation but cannot guarantee model accuracy in unfamiliar action regions.

Policy improvement in the learned simulator. Model generates "imagined trajectories"; policy and value functions undergo RL inside the simulator. Training bears the rollout cost; deployment outputs actions directly. If the model is differentiable, gradients can be used, but differentiability is not required for this path. A critical addition: video prediction models do not automatically yield reliable task rewards; the system still needs external rewards, learned reward models, or task evaluators. Policies may exploit model bugs, achieving high simulated return but failing in reality, so simulated return cannot be equated with true capability.

Convert predicted future into action. If the model only generates future observations, an inverse dynamics model can infer the required action from current and next observation; it can also generate pseudo-action supervision for action-less video. This provides an interface connecting passive video to control. However, predicted futures may not be executable; the same state change may correspond to multiple actions. Latent actions may not map directly to physical control commands; alignment with action semantics and execution mechanisms is still needed. When training inverse dynamics, one must avoid the latent representation encoding the to-be-predicted action prematurely, otherwise label leakage occurs instead of learning the true state-change-to-action relationship.

3.3 World-Action Models: Joint Learning of World Model and Action Policy

World prediction and action prediction can form a joint distribution. The paper uses the probability chain rule to show two factorizations:

Action first, then consequence:

p(oₜ₊₁, aₜ | hₜ, g) = π(aₜ | hₜ, g) · p(oₜ₊₁ | hₜ, aₜ, g)

Future first, then infer action:

p(oₜ₊₁, aₜ | hₜ, g) = p(oₜ₊₁ | hₜ, g) · p(aₜ | hₜ, oₜ₊₁, g)

The first corresponds to policy + forward dynamics; the second to future prediction + inverse dynamics. Both are probabilistic identities; the factorization alone does not imply identification of the causal effect of actions. If conditional probabilities are learned by separate models, consistency is not automatically guaranteed. Engineering implementations may use shared backbones with different heads, cross-modal attention, or unified generative models. Whether structure is shared and whether a consistent joint distribution is truly learned are separate questions; attention masks alone do not determine full probabilistic semantics — training objectives and generation processes must be considered. A motivation for joint modeling is letting visual changes provide richer supervision than action labels alone, but more pixels do not guarantee more effective control information; training must balance objectives to avoid visual details dominating learning. Another key distinction: joint training does not require joint generation at deployment. Models can learn world changes and actions simultaneously during training, but at inference only output actions to reduce latency. Evaluation should separate the benefit of auxiliary prediction supervision from the benefit of explicit rollout at inference, not conflate them.

3.4 Language Models as Implicit World Models and Generalist Agents

Language models learn sequence regularities by next-token prediction. Text contains environment structure, action experience, and interaction knowledge, so models may form implicit representations of some world aspects. However, "generating plausible narratives" differs from "having reliable, intervenable dynamics." Serializing task, observation, and action sequences allows a single autoregressive framework to play different roles: predicting action after history → policy; predicting new observation after action → forward prediction; generating future then outputting action that leads to it → future prediction + inverse dynamics. Role differences stem from conditioning and prediction targets, not requiring entirely different networks. Research on chess state representations shows models on specific sequence tasks may develop internal representations related to hidden states. Such evidence is suggestive but cannot be directly generalized to reliable understanding of arbitrary real environments, novel interventions, and long-term consequences. The key question is how to connect language pretraining knowledge with interaction data. One route trains an external environment model for the language agent to plan in; another jointly learns prediction, action, and feedback inside the policy. Regardless of structure, one must examine whether the model can correct errors in real interaction, not judge world knowledge validity solely by linguistic fluency.

4. Discussion and Conclusion

The paper concludes with five discussion directions that serve as a checklist for future world model development:

Cross-embodiment transfer: How to divide shareable knowledge from body-specific interfaces? Objects and events in a scene may share structure, but different robots have different sensors, joint structures, and action ranges. Cross-environment generalization ≠ cross-embodiment generalization. General world knowledge must combine with embodiment-specific observation encodings, action mappings, and execution constraints; a policy effective on one hardware cannot be directly viewed as universal control capability.

Scaling learning: Expand both data sources and supervisory signals. Images, general video, first-person recordings, robot trajectories, code, and web interactions contain diverse experience forms. Scaling is not just more samples but also extracting multiple signals (action prediction, future prediction, state change, inverse dynamics) from the same trajectory. They are complementary, but more supervision does not automatically mean more transferable knowledge; data and objectives must align with decision needs.

Representation and abstraction: What states to select, at what granularity to describe actions? Actions can be torques, joint commands, end-effector poses, or high-level skills. Abstraction reduces prediction and planning difficulty but may lose information needed for fine-grained control. When a world model predicts high-level skill effects, it often learns the closed-loop dynamics formed by environment and low-level controller together, not the bare environment response to raw control inputs. Changing the low-level controller may change the model's applicability.

Reasoning and planning: How does language-level task decomposition connect to executable predictions? High-level reasoning excels at goal decomposition, option comparison, and proposing intermediate steps; prediction models provide grounds for action consequences. Open question: in what space should they collaborate — natural language, structured physical state, latent representation, or hybrid? Can extra reasoning computation improve long-horizon decisions while meeting real-time control demands? Requires validation in concrete scenarios.

Evaluation: Focus is not on visual fidelity but on decision effectiveness. Prediction accuracy is a natural metric but not the ultimate one. A model may accurately reconstruct background textures yet miss contact relations critical for grasping; another compact model may fail to generate high-fidelity images but retain task-completion information. Small errors in regions the policy actually visits may matter more than large errors in irrelevant regions. Evaluation should simultaneously consider prediction fidelity, planning performance, policy improvement results, and cross-task/cross-environment generalization. Whether high-fidelity simulation necessarily yields better policies is posed as a research question, not a settled conclusion.

The overarching judgment: The value of a world model lies in organizing experience into knowledge that supports selection, prediction, and correction of actions. When evaluating a method, one can sequentially ask: What does it predict? What information is given? What decision-relevant content does the representation retain? How do predictions enter the action loop? These questions are more discriminating than simply judging whether a model "looks like the world."

Paper reference: Zhihui Zhu, An Introduction to World Models for Generalist Agents , arXiv:2610.30789.