Understanding MolmoAct: The Next‑Generation Large Action Model for Robotics

Large language models (LLMs) have evolved into large reasoning models, and the field is now moving toward large action models that can generate executable physical actions. The article cites Nvidia CEO Jensen Huang’s mention of "large action models" and argues that learning actions from data is difficult because most knowledge about physical tasks comes from embodied experience rather than text.

Current robot foundation models often map perception directly to control signals, limiting adaptability and semantic understanding. Action Reasoning Models (ARMs) aim to bridge this gap, and MolmoAct is presented as a state‑of‑the‑art ARM that encodes observations and language commands into deep perception features, produces an editable trajectory representation, and finally outputs precise low‑level action commands, enabling interpretability and intervention.

The model first uses an autoregressive predictor to encode multimodal observations and natural‑language instructions into a structured 2.5‑D representation (deep perception feature identifiers). These identifiers trigger the generation of intermediate planning representations, visualized as trajectory maps that guide the robot’s low‑level actions.

Key technical insights include the use of depth estimation to provide millimetre‑level spatial accuracy even in low‑light conditions, and a novel tokenization scheme that replaces random vocabulary symbols with the last 256 byte‑level BPE tokens from the Qwen2 tokenizer. This monotonic mapping preserves the geometric relationships of action intervals, improving initialization, embedding smoothness, and training efficiency.

Training data comprises 26.3 million samples, including 10.5 M robot action‑reasoning samples (RT‑1, BridgeData V2, BC‑Z), 1.5 M depth samples, 1.5 M trajectory samples, and 2 M multimodal web pages. Pre‑training uses 256 H100 GPUs (batch 512) for 100 k steps (~9,728 GPU‑hours). Mid‑stage training adds 1 M action‑reasoning and 1 M trajectory‑conditioned samples on 128 H100 GPUs (batch 128) for 50 k steps (~2,304 GPU‑hours). Post‑training adapts to new tasks with 30–50 demonstrations per task, action chunking (N = 8), LoRA fine‑tuning (rank 32, α = 16), and batch sizes of 128 (simulation) or 64 (real world).

Performance results show MolmoAct‑7B achieving higher training efficiency than many VLA models while maintaining or surpassing their accuracy. On the Simpl​erEnv benchmark, MolmoAct attains a 72.1 % out‑of‑distribution success rate, beating models from Physical Intelligence, Google, Microsoft, and NVIDIA. In the LIBERO suite, it reaches an 86.6 % average success rate, the best among major labs, and demonstrates strong parameter‑efficient fine‑tuning.

Real‑world experiments introduce varied perturbations (instruction rewrites, novel objects) and reveal that MolmoAct consistently outperforms OpenVLA and π0 Fast, confirming its superior generalization.

Appendix links provide access to the model checkpoint (https://huggingface.co/allenai/MolmoAct-7B-D-0812), the arXiv paper (https://arxiv.org/pdf/2508.07917), the official blog (https://allenai.org/blog/molmoact), the source code repository (https://github.com/allenai/MolmoAct), and the dataset (https://huggingface.co/datasets/allenai/MolmoAct-Dataset).