Can World Models Bridge LLMs' Dynamic Reasoning Gaps?

Dynamic Decision‑Making in LLM‑Based Agents

LLM agents deployed for web navigation, tool invocation, code execution, or long‑horizon planning must reason about an environment whose state evolves after each action. Static benchmarks ignore these state transitions, leading to systematic over‑optimism. Studies show that introducing interruptions or context changes can degrade performance by up to 60 % on math and code tasks [2], and that static evaluations miss error accumulation that occurs in dynamic settings [1].

Limitations of Text‑Only Reasoning Optimizations

Industry methods such as Chain‑of‑Thought (CoT), Self‑Consistency, Tree‑of‑Thought (ToT) and LATS improve the textual reasoning path but remain confined to the text space. CoT’s linear reasoning exhibits structural limits and underperforms direct answering across model scales and benchmark complexities [3][4]. ToT and LATS enhance path selection but do not model environment state transitions, limiting their effectiveness for irreversible actions like web submissions or API calls [5].

World‑Model Augmentation

World models learn a mapping from actions to subsequent environment states, giving agents predictive capability before execution. Representative systems integrate world models in distinct ways:

EvoAgent introduces a continuous world model that enables self‑planning and self‑reflection in open worlds. On Minecraft and Atari, it raises average success rates by 105 % and reduces invalid actions by more than sixfold compared with prior methods [6].

WebEvolver co‑evolves a world model with a web‑agent framework; forward‑simulation during inference guides action selection, yielding a 10 % performance boost on real‑web benchmarks such as Mind2Web‑Live and WebVoyager [7].

COMAP closes the loop between a text‑based world model and the agent policy. The model predicts future states for candidate actions, the agent optimizes actions accordingly, and resulting trajectories are used to distill the world model. On Qwen‑3‑4B, COMAP achieves a 16.75 % relative improvement across embodied planning, web navigation, and tool‑use benchmarks [8].

RWML learns an action‑conditional world model by aligning simulated next‑state predictions with real observations (sim‑to‑real gap reward). On ALFWorld and τ² Bench, it raises RL scores by 6.9 and 5.7 points respectively over baselines that use only task‑success rewards [9].

ProPlay abstracts successful trajectories into programmable graphs, allowing agents to pre‑play future program paths. Experiments show consistent superiority over strong baselines in environment understanding and self‑evolution [10].

How World Models Improve Agent Capabilities

Inference phase: The world model predicts the downstream state for each candidate action; the agent can verify and filter actions based on these predictions. This mechanism underlies WebEvolver’s 10 % gain on live web tasks [7].

Training phase: The world model serves as a virtual environment that generates interaction trajectories or simulates user feedback, reducing reliance on costly real‑world data. Joint optimization of the world model and policy mitigates distribution shifts between training and deployment environments [8][9].

Code example

③ WebEvolver 将协同进化的世界模型引入 Web Agent 框架，在推理阶段通过前瞻模拟指导动作选择。在 Mind2Web-Live、WebVoyager 等真实网页环境中，该方法相较现有自进化 Agent 取得了 10% 的性能提升。[7]
④ COMAP 通过闭环交互让文本世界模型与 Agent 策略协同进化。世界模型预测候选动作的未来状态，Agent 据此优化动作，生成的轨迹再通过自蒸馏更新世界模型。在具身任务规划、Web 导航和工具使用基准上，COMAP 在 Qwen3-4B 上实现了 16.75% 的相对提升。[8]
⑤ RWML 通过 sim-to-real gap 奖励在文本状态上学习动作条件世界模型，将模拟的下一状态与真实观测对齐。在 ALFWorld 和 τ² Bench 上，RWML 结合任务成功奖励后分别比直接使用任务成功奖励的 RL 高出 6.9 和 5.7 个点。[9]