How DM0.5 Brings Zero‑Shot, Long‑Term Memory, and Robustness to Real‑World VLA

Problem Context

Visual‑language‑action (VLA) models have demonstrated the ability to follow language commands, recognize objects, and execute robot actions in controlled laboratory settings. When deployed in open environments with variable lighting, camera drift, and unpredictable human interference, these models often fail to generalize.

DM0.5 Core Improvements

The DM0.5 embodied foundation model introduces five systematic upgrades aimed at real‑world generalization:

Emergence of zero‑shot capabilities.

More efficient and reliable fine‑tuning.

Continuous memory up to roughly 60 seconds.

Increased robustness to disturbances.

Cross‑platform transfer across diverse robot bodies.

Data‑Centric Engineering

Training data combine robot tele‑operation logs, embodied navigation recordings, first‑person human demonstrations, and general multimodal vision‑language corpora. The dataset spans multiple robot platforms, including ALOHA, Galaxea R1 Lite, AgiBot G1, Franka Emika Panda, UR5, ARX5, and Dexmal’s dual‑arm mobile robot.

Four cleaning stages ensure high‑quality supervision:

Removal of clearly erroneous recordings (e.g., physical discontinuities, mismatched image‑state pairs).

Discarding low‑information static frames.

Filtering low‑value actions that do not contribute to task learning.

Unifying action representations to handle redundant degrees of freedom across robots.

An automated relabeling pass aligns multimodal annotations with the true execution flow.

Architectural Enhancements

Long‑term memory : A Context Abstraction Layer compresses several seconds of visual history into a “history token” concatenated with the current observation. The model can ingest up to ~60 seconds of past frames; training uses random history lengths and augmentation to retain performance when history is absent.

Embodied reasoning : Eleven self‑regressive reasoning tasks are added during training, covering task planning, environment prediction, and action‑intent inference. The model predicts not only the next motion segment but also the current task stage, upcoming events, and the role of the ongoing action.

Trajectory alignment : A Trajectory Alignment Layer matches each predicted motion segment to a monotonic anchor point on the ground‑truth trajectory, allowing variable demonstration speeds while preserving action order and continuity.

Real‑World Evaluation

Zero‑shot tests assembled eight basic action primitives and seven semantic constraints on three platforms (Franka π0.5‑Droid, DM0.5‑Droid, Dexmal‑Mirror). DM0.5 consistently outperformed its predecessors on most action‑condition pairs, indicating the ability to compose unseen action‑language combinations.

Fine‑tuning on the RoboChallenge Table30 v2 benchmark—covering long‑term memory, multi‑step sequencing, precise grasp‑place, tool use, and bimanual coordination—achieved a 42 % overall success rate and a composite score of 61.

Memory experiment required the robot to recall the initial cup position after cleaning a table; the 60‑second history token successfully constrained later placement actions.

Robustness test evaluated nine camera‑pose variations on the Franka platform. Success‑rate fluctuations were minor, and the model exhibited a two‑stage strategy: a global coarse positioning followed by wrist‑level refinement. The model also recovered gracefully from human‑induced object moves or brief occlusions, adjusting motions based on the updated visual state.

Open‑Source Resources

GitHub repository: https://github.com/dexmal/opendm

Hugging Face model hub: https://huggingface.co/Dexmal/DM05