StepOPSD: Precise Step‑Level Error Detection for Multi‑Turn Agent RL

Agent reinforcement learning (RL) suffers from sparse rewards: a terminal reward only indicates success or failure of an entire multi‑turn trajectory, providing no signal about which token or step caused an error.

StepOPSD overview

Step‑Aware Online Preference Distillation (StepOPSD) adds a post‑hoc “review” step after a GRPO rollout. It (1) splits the full trajectory into action‑centered steps, (2) uses a hindsight teacher derived from a successful rollout in the same batch, and (3) converts the teacher‑student log‑probability gap into a multiplicative weight for the advantage while preserving the advantage’s sign.

Δ = log p_teacher(token | hindsight context) - log p_student(token | original context)

w_raw = 2 × sigmoid(sign(A) × Δ)
w = clip(w_raw, 1 - α_clip, 1 + α_clip)
 = (1 - λ_mix) × A + λ_mix × w × A

Two knobs control the process: λ_mix: proportion of teacher signal mixed into the advantage. α_clip: per‑token correction magnitude cap.

Key design components

1. Extract only controllable steps

For embodied tasks (e.g., ALFWorld) the action_only filter keeps only actions, discarding observations. For search‑QA tasks the clean_step_no_observation filter retains reasoning and query tokens while masking external retrieval results, ensuring supervision is not wasted on uncontrollable tokens.

2. In‑place teacher from successful rollouts

The teacher requires no external gold trajectory. It uses (a) the binary success/failure label of the current rollout and (b) a hindsight teacher taken from a successful rollout within the same GRPO batch. For each token the gap Δ is computed as above. A large positive Δ indicates the teacher prefers that token, highlighting a potentially critical decision point; a large Δ with opposite RL sign signals a conflict.

3. Adjust advantage magnitude without changing direction

The gap Δ is transformed into a weight w (clipped by α_clip) and blended with the original advantage A using λ_mix. This preserves the sign of A, so the policy’s direction (toward or away from a state) remains unchanged while the magnitude is locally re‑weighted.

Experimental setup

Experiments were conducted on ALFWorld (embodied) and Search‑QA (retrieval) using Qwen‑3‑1.7B and Qwen‑2.5‑3B‑Instruct. Training ran for up to 150 steps; the teacher was refreshed every 10 steps.

Results on ALFWorld

1.7B, λ_mix=0.05: Heat 60.9% (vs GRPO 40.0%, SDAR 33.3%).

1.7B, λ_mix=0.05: Pick2 55.0% (best at this scale).

3B, λ_mix=0.2, α_clip=0.05: Heat 79.1%, Cool 78.9%, Pick2 95.0%.

3B, λ_mix=0.2, α_clip=0.05: ALFWorld average 83.6%.

Simple sub‑tasks (Pick, Look) show little benefit, while tasks requiring precise state transitions (Heat, Cool, Clean) gain substantially because a single missed action drags the entire reward to zero.

Results on Search‑QA

3B, λ_mix=0.05, α_clip=0.05: average 45.7% (3B best).

3B, λ_mix=0.05, α_clip=0.05: NQ 45.0%, TriviaQA 61.6%.

1.7B, λ_mix=0.05: PopQA 45.6% (best at this scale).

1.7B/3B, λ_mix=0.2: HotpotQA 37.1% / 40.4%.

Datasets with sensitive query wording (TriviaQA, PopQA, HotpotQA) benefit more from step‑level shaping than simpler NQ, where a single accurate search often suffices.

Analysis of teacher‑student gap variance

The standard deviation of Δ (Std(Δ)) was tracked as a stability metric. Around step 50, λ_mix linearly decays to zero, ending explicit shaping; thereafter the gap variance reflects drift between the mature policy and the stale teacher.

Higher λ_mix in sparse‑reward Search‑QA stabilizes the gap variance, preventing policy drift, while in ALFWorld excessive shaping can clash with exploration, so a milder λ_mix is preferable.

The clipping parameter α_clip acts as a safety valve: tighter clipping (α_clip = 0.05) reduces the variance of advantage updates (e.g., from 0.770 to 0.252) and prevents a few tokens with extreme corrections from hijacking the whole update. It also shortens the average response length in Search‑QA while increasing tool‑use frequency.

Practical take‑aways

Step‑aware credit assignment, not reward sparsity, is often the bottleneck in multi‑turn agent tasks.

StepOPSD provides a lightweight, post‑hoc correction that requires no extra value model, no online teacher, and preserves the sign of the RL advantage.

Adjusting λ_mix and α_clip per environment balances shaping strength and stability.