LingBot-Video: The Open‑Source Video Foundation Model Tailored for Robots

Motivation and Problem Statement

Current text‑to‑video models such as Sora or Veo excel at producing high‑quality, aesthetically pleasing videos, but they often violate basic physical laws—e.g., water hovering in mid‑air or drawers opening without contact. For robots, such errors become harmful because a model could misinterpret a generation mistake as a true physical rule, leading to incorrect action learning.

LingBot-Video Overview

Ant RoboWave (蚂蚁灵波) released LingBot-Video , an open‑source video foundation model specifically designed for embodied intelligence. Unlike generic video models that focus on visual style and motion, LingBot-Video emphasizes actions, tasks, interactions, and physical environment changes to serve world‑prediction and robot‑training needs.

Three‑Level System Design

Architecture : Introduces a sparse Mixture‑of‑Experts (MoE) video diffusion framework that enables dynamic sparse activation, expanding model capacity while keeping inference cost low.

Data : Constructs a robot‑enhanced pre‑training corpus that merges internet‑scale video with robot manipulation, navigation, and first‑person perspective datasets, injecting explicit embodied priors.

Training : Develops a multi‑dimensional reward system that adds physical plausibility and task‑success signals to the usual aesthetic objectives.

Architectural Details

The core uses a single‑stream Diffusion Transformer (DiT) where visual tokens and condition tokens share the same sequence. Sparse MoE layers replace dense feed‑forward networks, routing each token to a subset of experts. This design provides two benefits: (1) a ten‑fold increase in total parameter capacity for richer physical priors, and (2) constant per‑token compute, preventing inference cost from scaling linearly with model size.

Experts are divided into shared (general knowledge) and routing (task‑specific) groups, following the DeepSeekMoE approach, which reduces task interference and encourages specialization.

Scaling Experiments

Comparisons between MoE 13B‑A1.4B and a dense 1.3B baseline show lower training and validation loss for the sparse model. Larger MoE variants (30B‑A3B, 60B‑A6B, 120B‑A11B) consistently outperform dense models of comparable activation size, demonstrating that capacity‑compute decoupling yields better scaling efficiency.

Inference latency tests on token sequences up to 1 M tokens reveal that MoE models achieve speed‑ups of 1.5×–3.2× over dense counterparts with similar activation budgets, confirming that routing overhead is minimal.

Data Curriculum

The data pipeline consists of five stages:

Train on 192p images to learn basic visual priors.

Introduce 192p video plus >70 k hours of embodied data (robot manipulation, navigation, first‑person views).

Upgrade to 480p images and video, emphasizing high‑motion content.

Perform source‑aware rebalancing at 480p, preserving more embodied samples.

Fine‑tune a small 1080p subset for the refiner to add high‑resolution details.

This curriculum injects robot‑action, physical interaction, and long‑tail motion data early, giving the model strong embodied priors.

Multi‑Dimensional Reward & RL

After pre‑training, LingBot-Video is aligned using a six‑component reward suite and Group Relative Policy Optimization (GRPO):

Visual quality – penalizes blur, artifacts, low resolution.

Text‑video alignment – matches structured captions to generated actions.

Dynamic intensity – encourages sufficient motion.

Motion coherence – rewards natural speed and rhythm.

Human pose consistency – avoids impossible body configurations.

Physical plausibility – enforces object permanence, non‑penetration, realistic material behavior.

GRPO maximizes these rewards, shifting the model from merely “pretty” video generation to physically trustworthy embodied video synthesis.

Evaluation

Internal benchmarks assess two dimensions: General Quality and Embodied Domain performance, across Text‑to‑Video (T2V) and Text‑and‑Image‑to‑Video (TI2V) settings. LingBot-Video ranks first among five open‑source competitors (NVIDIA Cosmos 3, Wan 2.2 A14B, LongCat‑Video, Hunyuan Video 1.5, LTX‑2.3) on TI2V embodied scores and second on T2V general quality, demonstrating robust physical reasoning even without an initial image.

Further tests on EgoDex Eval and DreamDojo‑HV Eval (A2V) show strong out‑of‑distribution action following, confirming that the model can generalize beyond training trajectories.

Applications in Robotics

When equipped with physical prediction ability, LingBot-Video can serve three roles in robot pipelines:

Data Engine : Generate additional training samples to augment costly real‑world robot data.

Policy Evaluator : Simulate outcomes of candidate actions for safety‑first pre‑deployment checks.

Action Planner : Produce concrete motion trajectories, acting like a visual “GPS” for robots.

Thus the model transitions from a content‑creation tool to a core component of a robot’s world model.

Conclusion

LingBot-Video demonstrates that an open‑source, MoE‑based video foundation model can be systematically engineered—through architecture, data curriculum, and multi‑dimensional RL—to deliver physically consistent video generation useful for embodied AI. While challenges such as long‑sequence consistency, fluid dynamics, and closed‑loop integration remain, the release provides a reproducible baseline that lowers the research barrier for future embodied video models.