Value as Memory (VaM): Robots Achieve 95%+ Success in 4-Hour Continuous Tasks
VaM uses recent value predictions as task progress memory for robot policies, combined with a four-stage offline-to-online RL pipeline, enabling long-horizon deformable object manipulation and contact-rich assembly with 95%+ success over 4 hours, 29.4% improvement from value caching, and 32.8% higher sample efficiency from online RL.
Value as Memory (VaM): Task Progress Memory for Robot Policies
Value as Memory (VaM) introduces a mechanism that treats recent value predictions as "task progress memory" and feeds them directly into the action policy. This memory helps robots distinguish visually similar states that require different actions — such as whether a gripper has just lost contact with a garment or has re-grasped it — and supports continuous adaptation through an offline-to-online reinforcement learning loop. In real-world experiments, VaM achieves over 95% success rate across 4-hour autonomous runs on three long-horizon tasks: Box Assembly, Laundry Folding, and Magnetic Bus Assembly.
Why Robots Need Task Progress Memory
Two fundamental challenges hinder reliable deployment of general robot policies:
Partial observability : Cameras provide limited viewpoints, and observation histories have fixed length. Visually similar frames can correspond to different underlying states (e.g., "about to latch" vs. "already stuck" in box assembly), requiring different actions.
Delayed deployment feedback : Serial data collection and training cycles slow iteration. Purely on-policy training wastes prior demonstrations and correction data, reducing sample efficiency.
VaM addresses both by augmenting the policy with recent value predictions (progress memory) and establishing a concurrent online RL framework that continuously incorporates new experience while retaining old data.
Value Caching: Turning Value Predictions into Policy Memory
In reinforcement learning, the value function estimates future return. VaM repurposes a sequence of recent value predictions as a compact signal of task progress — advancing, stalling, or regressing. This value caching mechanism feeds the cached value history into the policy alongside visual history and proprioception.
Implementation details:
The action expert is built on the π₀.₅ architecture.
A value model predicts the return distribution.
A GRU encodes the history of value predictions and proprioceptive state.
Value prediction and action generation run asynchronously: the policy reads already-computed historical values, avoiding latency.
Four-Stage Training Pipeline: From Demonstrations to Autonomous Experience
VaM organizes training into four progressive stages, each expanding the data distribution and autonomy level:
Stage 1: Offline Demonstration Initialization — Expert demonstrations initialize both action and value experts, providing basic manipulation skills and initial progress estimation.
Stage 2: Offline RL Warm-up — Policy rollouts and human interventions are collected. The value model learns to recognize stalls, failures, and recoveries. Updated value predictions generate advantage labels for policy improvement.
Stage 3: Online DAgger — The robot executes tasks with human corrections allowed. New on-policy states are continuously gathered. Training uses offline demonstrations, historical interventions, and current trajectories in parallel, forming a real-time on-robot RL loop.
Stage 4: Learn from Experience (LfE) — The robot runs fully autonomously, learning from sparse task-outcome rewards without new human corrections. All prior data (demonstrations, interventions, online trajectories) remain in the training buffer.
Real-World RL Evaluation on Three Long-Horizon Tasks
Tasks cover deformable object manipulation and contact-rich assembly:
Box Assembly — Precise insertion and latching of cardboard flaps.
Laundry Folding — Manipulating flexible fabric with frequent contact loss and re-grasping.
Magnetic Bus Assembly — Aligning and snapping magnetic components.
Each task requires 10–20 minutes per episode; policies are evaluated over hundreds of episodes across training checkpoints.
Value Caching Boosts Success Rates by 29.4%
Ablation comparing full VaM against a variant without value caching (all other components identical) shows:
Across the last three online RL checkpoints, VaM improves final task success rate by 29.4% relative to the no-cache baseline.
The benefit persists after human corrections stop (LfE phase), indicating the memory generalizes to autonomous recovery.
Value Caching Improves Recovery from Failures
Beyond final success, the paper measures recovery rate : the fraction of stalled task phases where the robot executes corrective actions and completes the phase. Blind evaluation of LfE checkpoints yields:
Box Assembly: VaM 54.5% vs. no-cache 50.0%
Laundry Folding: VaM 86.4% vs. no-cache 58.8%
Magnetic Bus Assembly: VaM 33.3% vs. no-cache 7.4%
The larger gaps on Laundry Folding and Magnetic Bus Assembly suggest value caching is especially helpful when visual ambiguity is high (fabric folds, magnetic pre-contact). Qualitative trajectories show VaM's value predictions and policy representations (PCA projections) clearly reflect stagnation and subsequent correction, while the baseline's representations remain confused.
Effective Value Caching Requires Up-to-Date Value Estimates
To test whether the cache itself or its content matters, the authors freeze the trained action expert and swap the value model with earlier versions (demonstration-initialized or offline-warmed). The caching mechanism stays identical, but success rates drop sharply on all three tasks.
In a finer-grained Laundry Folding analysis, with action expert and all observations fixed, only the value history is replaced. After losing contact, the current value history guides the policy to open and reposition the gripper for re-grasp. After re-grasping, the same history keeps the gripper closed. An older value model incorrectly predicts opening at the end of the sequence, causing failure.
Online RL Improves Sample Efficiency by 32.8% and Reduces Real-World Time by 57.8%
Comparing online RL (VaM's concurrent training) against an offline RL baseline matched on gradient steps and data collection budget:
At the end of LfE, online RL achieves 32.8% higher average success rate across the three tasks.
Concurrent execution of data collection, value training, and policy training reduces real-world time per gradient step by 57.8% compared to the sequential offline pipeline.
Conclusion
VaM demonstrates a practical use of value models: recent value predictions serve as task progress memory, directly informing action selection. Coupled with a four-stage offline-to-online real-robot RL pipeline, the system leverages historical demonstrations and interventions while continuously adapting to the states encountered by the current policy in the real world. The approach yields substantial gains in long-horizon success, recovery capability, and sample efficiency.
Paper: Value as Memory: Online Value-Guided Post-training for General Robot Policies . Project page: https://virlus.github.io/vam/. Code: https://github.com/Virlus/VaM.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
