Behavior Consistency Beats State Consistency in Text World Models for Agents

Text-based world models simulate environments for LLM agents to enable offline evaluation and lookahead planning without costly real-world interaction. Current approaches train these models to maximize textual similarity to real observations (state consistency) using metrics like F1, BERTScore, and ROUGE-L. However, the authors discover a metric reversal phenomenon: higher text fidelity does not guarantee better agent decisions, and can even degrade performance.

Metric Reversal: Text Fidelity ≠ Functional Correctness

Figure 1 illustrates two hallucination types in an e-commerce scenario. Output 1 omits the target sweater (BERTScore 0.974) causing agent failure; Output 2 omits two irrelevant items (BERTScore 0.859) yet the agent succeeds. Standard metrics penalize the benign omission more harshly than the fatal one. Controlled perturbation experiments confirm that F1, BERTScore, ROUGE-L, and even GPT-4o-as-judge all rank the worse output higher. Only the proposed BehR metric correctly orders them with a dense signal.

Analogy: Literal translation is not necessarily good translation; what matters is whether the reader can make the same judgment from the translation as from the original.

From State Consistency to Behavior Consistency

The authors define Functional Consistency: a predicted state is good if a frozen reference agent produces the same action distribution in the predicted state as in the real state. Formally, for real state s, predicted state ŝ, and history h, the goal is to minimize the divergence between π_ref(a|s,h) and π_ref(a|ŝ,h).

Method: Behavior Consistency Reward (BehR)

BehR uses a frozen reference agent to score the same action under both states. The reward is the negative absolute difference in log-probabilities: r = -|log π_ref(a|s,h) - log π_ref(a|ŝ,h)|. If the action scores match, reward is maximal; removing the target item causes a sharp score drop (low reward); removing irrelevant items barely changes scores (high reward). Training uses GRPO (Group Relative Policy Optimization): the world model generates multiple candidate predictions per sample, each receives a BehR score, and group-wise relative normalization yields the policy gradient.

Experimental Setup

Environments: WebShop (e-commerce navigation) and TextWorld (text adventure).

Base world models: Qwen2.5-7B and LLaMA3.1-8B.

Evaluation agents: Qwen3-8B, Qwen3-32B, GPT-4o, GPT-5 (4 agents).

Metric: Pairwise consistency rate CR_pw — whether the agent's task outcome in the world model matches the real environment (1.0 = perfect).

16 total configurations (2 bases × 2 envs × 4 agents).

Results

Across 16 configurations, BehR improves CR_pw in 13 settings and matches the other 3, with no regressions. Notable gains on WebShop:

Qwen3-8B: 0.345 → 0.483 (+13.8 percentage points, ~+40%).

GPT-4o: 0.760 → 0.840 (+8.0 pp, ~+10.5%).

On TextWorld, strong agents already near ceiling; BehR maintains near-perfect consistency.

Downstream Applications

Offline Evaluation

When the world model is simpler than reality, weak agents can falsely appear successful. On TextWorld, Qwen3-0.6B achieves 0% real success, but the baseline world model labels 42.5% of tasks as successful (false positives). BehR reduces false positives to 9.5%, raising consistency from 57.5% to 90.5%.

Lookahead Planning

Agents propose 5 candidate actions per step; the world model simulates each and picks the best. BehR helps Qwen3-8B on WebShop improve success from 14.5% to 25.5% (+11.0 pp), outperforming the baseline's +9.0 pp gain. Weaker agents benefit more because they rely more on simulation before acting.

Ablation and Open Resources

Ablation confirms gains stem from the reward function itself, not other training details. The paper, code, and models are openly available:

Paper: https://arxiv.org/abs/2604.13824 Code: https://github.com/Ricardo-H/behr-wm Models:

https://huggingface.co/collections/Ricardo-H/behr-behavior-consistent-world-models

Conclusion

The work reveals a fundamental misalignment in text-based world model training: optimizing for textual similarity pushes models toward fatal hallucinations. By switching the objective to behavior consistency via BehR, the model learns to make errors that are harmless to downstream agent decisions. This shift yields consistent improvements across diverse agents, environments, and base models, and directly benefits offline evaluation reliability and lookahead planning effectiveness.