Skip to content
Value as Memory (VaM): Robots Achieve 95%+ Success in 4-Hour Continuous Tasks

Value as Memory (VaM): Robots Achieve 95%+ Success in 4-Hour Continuous Tasks

VaM uses recent value predictions as task progress memory for robot policies, combined with a four-stage offline-to-online RL pipeline, enabling long-horizon deformable object manipulation and contact-rich assembly with 95%+ success over 4 hours, 29.4% improvement from value caching, and 32.8% higher sample efficiency from online RL.

Machine Heart
Machine Heart
Machine Heart

Value as Memory (VaM): Task Progress Memory for Robot Policies

Value as Memory (VaM) introduces a mechanism that treats recent value predictions as "task progress memory" and feeds them directly into the action policy. This memory helps robots distinguish visually similar states that require different actions — such as whether a gripper has just lost contact with a garment or has re-grasped it — and supports continuous adaptation through an offline-to-online reinforcement learning loop. In real-world experiments, VaM achieves over 95% success rate across 4-hour autonomous runs on three long-horizon tasks: Box Assembly, Laundry Folding, and Magnetic Bus Assembly.

Why Robots Need Task Progress Memory

Two fundamental challenges hinder reliable deployment of general robot policies:

Partial observability : Cameras provide limited viewpoints, and observation histories have fixed length. Visually similar frames can correspond to different underlying states (e.g., "about to latch" vs. "already stuck" in box assembly), requiring different actions.

Delayed deployment feedback : Serial data collection and training cycles slow iteration. Purely on-policy training wastes prior demonstrations and correction data, reducing sample efficiency.

VaM addresses both by augmenting the policy with recent value predictions (progress memory) and establishing a concurrent online RL framework that continuously incorporates new experience while retaining old data.

Value Caching: Turning Value Predictions into Policy Memory

In reinforcement learning, the value function estimates future return. VaM repurposes a sequence of recent value predictions as a compact signal of task progress — advancing, stalling, or regressing. This value caching mechanism feeds the cached value history into the policy alongside visual history and proprioception.

Implementation details:

The action expert is built on the π₀.₅ architecture.

A value model predicts the return distribution.

A GRU encodes the history of value predictions and proprioceptive state.

Value prediction and action generation run asynchronously: the policy reads already-computed historical values, avoiding latency.

VaM architecture: value caching feeds recent value predictions into the policy via GRU, combined with visual and proprioceptive inputs.
VaM architecture: value caching feeds recent value predictions into the policy via GRU, combined with visual and proprioceptive inputs.

Four-Stage Training Pipeline: From Demonstrations to Autonomous Experience

VaM organizes training into four progressive stages, each expanding the data distribution and autonomy level:

Stage 1: Offline Demonstration Initialization — Expert demonstrations initialize both action and value experts, providing basic manipulation skills and initial progress estimation.

Stage 2: Offline RL Warm-up — Policy rollouts and human interventions are collected. The value model learns to recognize stalls, failures, and recoveries. Updated value predictions generate advantage labels for policy improvement.

Stage 3: Online DAgger — The robot executes tasks with human corrections allowed. New on-policy states are continuously gathered. Training uses offline demonstrations, historical interventions, and current trajectories in parallel, forming a real-time on-robot RL loop.

Stage 4: Learn from Experience (LfE) — The robot runs fully autonomously, learning from sparse task-outcome rewards without new human corrections. All prior data (demonstrations, interventions, online trajectories) remain in the training buffer.

Four training stages connecting offline demonstrations to autonomous experience learning; value expert supplies both advantage for training and progress memory for acting.
Four training stages connecting offline demonstrations to autonomous experience learning; value expert supplies both advantage for training and progress memory for acting.

Real-World RL Evaluation on Three Long-Horizon Tasks

Tasks cover deformable object manipulation and contact-rich assembly:

Box Assembly — Precise insertion and latching of cardboard flaps.

Laundry Folding — Manipulating flexible fabric with frequent contact loss and re-grasping.

Magnetic Bus Assembly — Aligning and snapping magnetic components.

Each task requires 10–20 minutes per episode; policies are evaluated over hundreds of episodes across training checkpoints.

Value Caching Boosts Success Rates by 29.4%

Ablation comparing full VaM against a variant without value caching (all other components identical) shows:

Across the last three online RL checkpoints, VaM improves final task success rate by 29.4% relative to the no-cache baseline.

The benefit persists after human corrections stop (LfE phase), indicating the memory generalizes to autonomous recovery.

Online RL success rate curves for VaM, no value caching, no offline warm-up, and online RECAP across three tasks, spanning Online DAgger and LfE phases.
Online RL success rate curves for VaM, no value caching, no offline warm-up, and online RECAP across three tasks, spanning Online DAgger and LfE phases.

Value Caching Improves Recovery from Failures

Beyond final success, the paper measures recovery rate : the fraction of stalled task phases where the robot executes corrective actions and completes the phase. Blind evaluation of LfE checkpoints yields:

Box Assembly: VaM 54.5% vs. no-cache 50.0%

Laundry Folding: VaM 86.4% vs. no-cache 58.8%

Magnetic Bus Assembly: VaM 33.3% vs. no-cache 7.4%

The larger gaps on Laundry Folding and Magnetic Bus Assembly suggest value caching is especially helpful when visual ambiguity is high (fabric folds, magnetic pre-contact). Qualitative trajectories show VaM's value predictions and policy representations (PCA projections) clearly reflect stagnation and subsequent correction, while the baseline's representations remain confused.

Qualitative comparison in Box Assembly: camera view, PCA of value expert representation, and value predictions for VaM vs. no-cache.
Qualitative comparison in Box Assembly: camera view, PCA of value expert representation, and value predictions for VaM vs. no-cache.
Value predictions and policy actions during contact loss and re-grasp in Laundry Folding.
Value predictions and policy actions during contact loss and re-grasp in Laundry Folding.

Effective Value Caching Requires Up-to-Date Value Estimates

To test whether the cache itself or its content matters, the authors freeze the trained action expert and swap the value model with earlier versions (demonstration-initialized or offline-warmed). The caching mechanism stays identical, but success rates drop sharply on all three tasks.

In a finer-grained Laundry Folding analysis, with action expert and all observations fixed, only the value history is replaced. After losing contact, the current value history guides the policy to open and reposition the gripper for re-grasp. After re-grasping, the same history keeps the gripper closed. An older value model incorrectly predicts opening at the end of the sequence, causing failure.

Success rates with fixed action expert but different value model versions (demo init, offline warm-up, final).
Success rates with fixed action expert but different value model versions (demo init, offline warm-up, final).
After contact loss, current value cache leads to open-and-reposition action.
After contact loss, current value cache leads to open-and-reposition action.
After re-grasp, current value cache maintains gripper closure; older model predicts opening.
After re-grasp, current value cache maintains gripper closure; older model predicts opening.

Online RL Improves Sample Efficiency by 32.8% and Reduces Real-World Time by 57.8%

Comparing online RL (VaM's concurrent training) against an offline RL baseline matched on gradient steps and data collection budget:

At the end of LfE, online RL achieves 32.8% higher average success rate across the three tasks.

Concurrent execution of data collection, value training, and policy training reduces real-world time per gradient step by 57.8% compared to the sequential offline pipeline.

Online vs. offline RL: higher LfE success rates and lower wall-clock time per gradient step.
Online vs. offline RL: higher LfE success rates and lower wall-clock time per gradient step.

Conclusion

VaM demonstrates a practical use of value models: recent value predictions serve as task progress memory, directly informing action selection. Coupled with a four-stage offline-to-online real-robot RL pipeline, the system leverages historical demonstrations and interventions while continuously adapting to the states encountered by the current policy in the real world. The approach yields substantial gains in long-horizon success, recovery capability, and sample efficiency.

Paper: Value as Memory: Online Value-Guided Post-training for General Robot Policies . Project page: https://virlus.github.io/vam/. Code: https://github.com/Virlus/VaM.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

offline-to-online RLlong-horizon manipulationVaMdeformable object manipulationrobot reinforcement learningtask progress memoryValue as Memoryvalue caching
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.