A Future‑Predicting Critic Propels VLA Reinforcement Learning

Limitation of single‑frame critics

In many Vision‑Language‑Action reinforcement‑learning pipelines the critic evaluates state quality from a single image or a single‑frame VLM feature. Because robot control is a partially observable Markov decision process, a single frame cannot reliably indicate whether the arm is approaching or receding from the target, whether a grasp is stable, or whether an object is still moving.

Why adding more frames alone does not help

Three phenomena were observed when naïvely feeding history frames or a temporal transformer to the critic:

Performance can degrade when past observations are added without additional supervision.

A transformer without a world‑prediction objective fails to exploit the history.

Only when the model is trained to predict the next latent state does the history translate into better value estimates. The missing ingredient is a supervision signal that forces the model to understand dynamics rather than merely aggregating static features.

World Critic Model (WCM) architecture

WCM adopts a lightweight LeJEPA‑style architecture composed of four modules: an observation encoder, a world predictor, a value head, and a dynamics prediction head. The processing pipeline consists of four steps:

Encode recent K frames. Each frame is independently encoded into a latent vector using either a Vision Transformer or the VLM backbone of the underlying VLA model.

Condition on language. Task instructions (e.g., “place the carrot on the plate”) are encoded and injected into the latent sequence, allowing the same visual scene to receive different value interpretations.

Aggregate with a causal transformer. The language‑conditioned sequence is fed to a causal Transformer world predictor, producing a shared history representation that supports both value estimation and next‑state prediction.

Joint dual‑head learning. The Value Head outputs the current state value (action‑free), while the World Head, conditioned on the current action, predicts the next latent state. The total loss is

value regression loss + world prediction loss + SIGReg feature‑space regularization

.

Compatibility with on‑policy and off‑policy RL

WCM is inserted as a plug‑in critic without redesigning the RL algorithm. In on‑policy settings it supplies value estimates for PPO or Flow‑SDE; in off‑policy settings it works with AWR, RECAP, or other algorithms. Thus WCM upgrades the critic representation rather than tying to a specific policy.

Experimental validation

ManiSkill: a VLA model with zero prior exposure improved from 0.78% success to 98.7% after WCM‑guided RL, a 97.9‑point gain, and out‑of‑distribution success rose by 72.7%.

LIBERO‑Plus: using only one demonstration (One‑SFT) and ~250 RL steps, WCM surpassed the Full‑SFT baseline that uses 50 demonstrations.

MetaWorld and CALVIN: WCM consistently raised success rates and task‑completion metrics across these benchmarks.

Real‑world WidowX‑250S: seven tasks (grasp‑place, cloth folding, stove cleaning, dynamic sushi grabbing) were trained with ~107 M trainable parameters; training completed in under 30 minutes and yielded higher performance and smoother policies compared with a VLM‑only critic.

Value‑curve visualization

The repository provides a tool that overlays predicted value curves on real trajectories. Successful runs show a monotonic rise in value, while failure points cause rapid drops, confirming that the critic learns a continuous process‑level signal rather than a binary endpoint classifier.

Resources

Paper: https://arxiv.org/pdf/2607.29613

Code: https://github.com/sylvestf/WCM