Terminal-RSI: 27B Agent Rivals Frontier Models via Recursive Self-Improvement

Terminal-RSI: A Recursive Self-Improvement Framework for Terminal Agents

Tencent Hy Frontier, in collaboration with the National University of Singapore, introduces Terminal-RSI , a training framework that enables a 27B-parameter model to reach frontier-level performance on terminal agent benchmarks. The framework addresses three core challenges in terminal agent training: credit assignment in long trajectories, value drift during recursive training, and misalignment between local rewards and full task success.

Five-Phase Recursive Training Loop

The framework alternates between Recursive Self-Rewrite (RSR) and Recursive Policy Refinement (RPR) across five phases, starting from Qwen3.8-27B as the base model.

Phase 1 – Base: Establish terminal task execution and verification pipeline. Each task includes public instructions, an execution environment, and a protected verifier; only full verifier passage counts as strict success.

Phase 2 – RSR-1: Agent discovers successful trajectories in complementary execution frameworks. A planner converts these into reusable runbooks; a reviewer performs binary gating on structure, artifacts, and compatibility. Approved runbooks are re-executed in the target framework; only trajectories passing both the protected verifier and information-leakage screening enter the supervised training set. Only the Actor (policy) is updated; Critic (value) weights are frozen.

Phase 3 – RPR-1: Freeze RSR-1 Actor. Train a Warm-up Critic Model via one SWR epoch, then run four Critic-only calibration steps, followed by joint Actor-Critic optimization using Family Dense Reward (FDR) and PPO.

Phase 4 – RSR-2: Repeat discovery, rewriting, and verification to expand high-quality supervised data. Critic parameters from RPR-1 are retained (not re-initialized).

Phase 5 – RPR-2: Inherit the previous value model, perform four Critic-only calibrations, then joint training. Actor learning rate reduced from 5×10⁻⁷ to 1×10⁻⁷ (5× lower) while Critic learning rate stays at 1.5×10⁻⁵ to minimize policy-update shock to inherited value estimates. The final retained branch uses Success-Prioritized Reward (SPR) ; an FDR branch is also compared.

Reward Design and Credit Assignment

FDR calibrates semantic progress rewards per task family based on checklist size.

SPR separates strict success (reward ≥ 1) from bounded partial progress (reward < 1), prioritizing full task completion in reward ranking.

Turn-Boundary Skip-Observation GAE (TB-GAE) retains token-level value baselines but prevents observation text from becoming advantage-recursion nodes. It applies discount and trace decay across interaction turns, reducing credit-assignment dependence on observation length.

Benchmark Results on Terminal-Bench

Average success rate across four Terminal-Bench splits rises from 20.68% (Base) to 50.58% (Terminal-RSI-27B) , a +29.90 percentage-point gain. Stage-wise averages: 20.68% → 34.25% (RSR-1) → 44.50% (RPR-1) → 48.75% (RSR-2) → 50.58% (RPR-2).

Terminal-Bench 2.1: 86.52% (surpasses GPT-6 Sol 85.40% and Kimi K3 85.00%; close to GPT-5.6 Sol 87.30%).

Terminal-Bench 4.0: 19.84% (edges out Gemini 3.8 Flash 19.70% and DeepSeek V4 Pro 14.60%).

Terminal-Bench Hard: 79.00% (up from 54.00% at Base).

Terminal-Bench 3.0: 16.95%.

Gains are not uniform across stages or splits; Terminal-Bench 2.1 plateaus at the final stage, indicating RPR-2 adds more value on other splits.

Long-Horizon Task Evaluation (LHTB & SWR100)

LHTB requires fixing broken projects, handling multi-step dependencies, and passing hidden stress tests. Process reward rises from 0.21 (Base) to 0.37 (final), a 76.19% relative increase. Strictly completed tasks grow from 0/46 to 5/46. RSR-1 improves process reward (0.29) but yields 0/46 strict completions, showing local progress does not guarantee full solutions. RPR-1 achieves 2/46, RSR-2 reaches 4/46, final model 5/46.

SWR100 (scientific workflow reconstruction, no training-task overlap) shows generalization: success rate climbs from 3.00% (Base) to 6.00% (RPR-1) then plateaus, indicating diminishing returns on out-of-distribution tasks.

Key Takeaway

By alternating supervised rewriting (RSR) that distills verified successful behaviors with value-inheriting policy refinement (RPR) that optimizes via carefully designed rewards and TB-GAE, Terminal-RSI demonstrates that a 27B model can competitively match or exceed much larger frontier models on terminal agent benchmarks, while also making measurable progress on long-horizon execution contracts.