Causal Upgrade for Embodied World Models: CD‑LAM Improves Action Control with Only a Few Thousand Fine‑Tuning Steps

Background

Embodied world models predict future visual frames conditioned on robot actions, enabling planning and data generation. Latent Action Models (LAM) extract pseudo‑labels from video changes to pre‑train world models on large unlabeled video corpora, then fine‑tune with a small amount of real‑action data.

Causal analysis of LAM

Intervention tests altered the causal factor (action) while keeping other inputs fixed. In a static‑action test, the initial frame was fixed and all subsequent actions were set to zero; the baseline LAM still generated noticeable arm motion, indicating that its latent vectors were driven by visual cues unrelated to motion. A target‑action test replaced the action sequence with that from a different video while keeping the initial frame unchanged; the baseline output changed little, showing a failure to follow the new action. Further probing of the LAM encoder revealed strong responses to identical frames, pure camera motion, and scene context, confirming that background and scene information leaked into the latent action representation (a “scene shortcut”).

Action‑focused evaluation (FDCE)

Pixel‑level metrics such as PSNR correlate weakly with action error. The authors introduced Foreground‑Diffusion‑Control Error (FDCE): foreground regions (robot arm and manipulated objects) are segmented with SAM3 masks, foreground points are tracked across frames, and the Euclidean distance between generated and reference trajectories is computed to quantify action deviation. PSNR remains the metric for overall visual fidelity.

CD‑LAM design

CD‑LAM leaves the world‑model architecture, latent‑action dimensionality, and action interface unchanged. It cleans the LAM output in three stages: Debias LAM → Train world model → Incorporate real robot actions . Three additional training objectives are added during the debiasing stage:

Embodied‑centered reconstruction : a spatially weighted loss emphasizes foreground pixels (higher weight) and down‑weights background pixels, forcing the latent to explain motion of the arm and objects.

Action‑centered contrastive learning : latent vectors are projected, L2‑normalized, and paired with a binary label indicating whether two video clips share the same action primitive. A SigLip loss pulls together vectors from the same primitive and pushes apart vectors from different primitives.

Latent‑space calibration : a zero‑action penalty shrinks the norm of latent vectors produced from identical frames toward zero, anchoring static representations at the origin. Per‑dimension KL regularization prevents collapse of the latent space.

After debiasing, the cleaned latent vectors condition a pre‑training phase of the world model. A post‑training phase then injects real robot control commands via a lightweight adapter, preserving the original latent dimensionality.

Experimental results

Using 2 B and 14 B scale models, CD‑LAM reduced action error by 42 % and 26 % respectively (without real actions) and improved PSNR by 3.41 dB and 2.14 dB. In target‑action tests, error reductions of 21 % (2 B) and 34 % (14 B) were observed. After post‑training with real robot data, action error fell an additional 34.8 % (2 B) and 30.4 % (14 B), and PSNR rose 0.75–1.0 dB. Fine‑tuning required only 3 k–6 k steps (≈10 % of the baseline training budget) to match a 50 k‑step DreamDojo baseline.

Data‑efficiency experiments showed that training the debiasing stage with only one hour of video retained ~80 % of the gains obtained with 1 000 h of data, demonstrating that cleaner latent actions substantially reduce downstream adaptation burden.

Resources

Paper: https://arxiv.org/abs/2607.09185

Code: https://github.com/yufanwei/CD-LAM

Project page: https://yufanwei.github.io/CD-LAM-project-page/

Figures