VAST: Adaptive Horizon Selection for Long-Horizon RL via Value Stitching

Problem: Fixed Horizons Limit Long-Horizon RL

Traditional single-step TD learning propagates errors through bootstrapping, while fixed multi-step methods (n-step TD, Q-chunking) cannot adapt to the varying temporal structure of long-horizon tasks. The article argues that critical task transitions rarely align with pre-defined segment boundaries, and fixed horizons either blur short-term decisions or limit far-future evaluation.

VAST: Horizon-Adaptive Offline Policy Learning via Value Stitching

Researchers from Hong Kong University and Tsinghua University propose VAST (Horizon Adaptive Offline Policy Learning via Value Stitching), a framework that lets the agent jointly select a future sub-goal state and the time span to reach it, stitching together local returns and subsequent value estimates.

G-Function: Flexible Segment Valuation

VAST introduces an auxiliary G-function G(s, s⁺, k) that estimates the expected discounted return of transitioning from state s to a future state s⁺ in exactly k steps. This provides a flexible supervisory signal for any segment length.

Compositional Consistency Constraint

To generalize across time scales, VAST adds a compositional consistency constraint: the value of a long segment should equal the sum of its sub-segments' values. This encourages the G-function to respect temporal compositionality.

Stitching Bellman Operator and Stitching Policy

The core update uses a stitching Bellman operator that selects an optimal intermediate state s⁺ and horizon k to maximize the sum of the G-function estimate and the downstream value. The operator is proven to be a contraction, guaranteeing a fixed point. The stitching policy π_stitch(s⁺, k | s) is trained with advantage-weighted learning (giving higher weight to higher-value state-horizon pairs) and a diffusion/flow model for expressive power.

Policy Decomposition: Stitching Policy vs. Execution Policy

VAST separates concerns: the stitching policy proposes future sub-goals (s⁺, k), while a goal-conditioned execution policy π_exec(a | s, s⁺) produces actions to reach s⁺. Notably, the execution policy does not receive k as input, avoiding train-test mismatch because training k comes from suboptimal behavior data while test k comes from the optimized stitching policy. At test time, rejection sampling draws multiple candidate actions for the chosen sub-goal and a Q-function selects the highest-value action.

Experimental Evaluation on OGBench

Experiments cover 10 domains and 50 state-based tasks (block manipulation, puzzles, maze navigation). VAST is compared against 8 baselines: single-step TD (IQL, FQL, IFQL), fixed 5-step TD (FQL-n, IFQL-n), chunked value (QC), and generative value models (VALUE-FLOWS, FLOQ). Each method trains offline for 1M steps; evaluation uses 8 random seeds and 50 episodes per task.

Key Findings

VAST achieves leading performance, especially on multi-stage robotic manipulation tasks requiring complex planning.

Visualizations of the "block stacking" task show that predicted horizons are not mechanically stretched but align with atomic task intents (e.g., reaching, grasping, placing). The stitching policy produces a coherent sequence of intermediate goals that decompose the distant final goal into executable steps.

Adaptive horizons enable the policy to judge "where to go next" at each stage, leading to stable, value-maximizing action selection.

Limitations and Future Directions

The authors acknowledge that current results are limited to state-based offline benchmarks. Extension to visual inputs, real robots, and more open environments remains untested. However, the work suggests a promising direction: instead of a fixed stride, models can learn when to subdivide and when to leap, letting local experience more effectively support long-term goals.

Paper: https://arxiv.org/pdf/2606.21136 (arXiv:2606.21136)<br/>Project page: https://whiterrrrr.github.io/value-stitching/