Lego‑RL Enables Stable, Reliable RL Training for Coding Agents Without SDK Modifications

TL;DR

Coding‑agent reinforcement learning faces three core challenges: (1) the harness rewrites dialogue history, allowing the model to cheat; (2) training failures can be caused by strategy degradation or infrastructure faults; (3) reliable reward signals are hard to obtain. Lego‑RL tackles these issues with three pillars—Faithful, Reliable, and Observable.

Why Train on Native Agent Harnesses?

Typical pipelines rewrite the agent to fit a training framework, changing the control flow and causing the learned policy to be optimal only for the modified harness. By keeping the original harness unchanged, Lego‑RL ensures that the learned strategy is directly applicable to the deployment environment.

Benchmark Improvements

Using the same Qwen3.5‑35B‑A3B base model, Lego‑RL raises SWE‑bench Verified scores on three harnesses from 64.0 / 62.4 / 57.2 to 70.4 / 68.2 / 66.6, outperforming a next‑generation Qwen3.6‑35B‑A3B upgrade (which adds only 3.4 / 1.0 / 3.4 points). This demonstrates that a well‑designed RL framework can yield larger gains than merely switching to a newer base model.

Core Features

Faithful (保真优化) : A process‑level proxy records token‑level IDs, log‑probs, and MoE routing during inference, then aligns them with the training context. Replay of MoE routing improves log‑prob consistency from 0.9946 to 0.9993.

Reliable (可靠执行) : Rewards come directly from task‑specific test scripts (binary 0/1). Anti‑cheat measures are placed inside the sandbox, preventing network‑based shortcuts, git‑history leaks, and test‑file tampering. Task difficulty is filtered so that each batch contains informative rollouts.

Observable (可观测训练) : Live UI visualizes per‑trial termination reasons, reward distributions, and training‑inference log‑ratio consistency, allowing rapid attribution of failures to strategy or infrastructure.

System Architecture

The trainer is built on verl for distributed actor learning (FSDP / Megatron‑LM) and on Harbor for sandboxed execution. An AgentLoopWorker launches an unmodified harness inside a per‑trial sandbox, routes all model calls through an in‑process proxy that supports both OpenAI and Anthropic APIs, and captures token‑level metadata for faithful training.

Key components:

Trainer (verl) : Handles actor updates, PPO/GRPO/GSPO objectives, rollout pools, and synchronous/asynchronous scheduling.

Sandbox (Harbor) : Packages the task description, repository snapshot, and verifier into a uniform format, runs the agent, collects binary rewards, and isolates the environment.

AgentLoopWorker : Manages the end‑to‑end trial lifecycle, keeping GPU usage low (≈9% of time) while the agent executes the majority of the workload.

Training Process and Optimizations

Three production runs (3 epochs, 126 steps each) on 2,699 tasks with 200 k context show steadily rising reward curves and stable policy entropy. The harness continually rewrites its history, but the proxy‑recorded transcripts keep training stable.

Task difficulty filtering ensures that each batch contains rollouts with mixed success (1–3 successes out of 8 attempts), preventing zero‑variance groups that would provide no gradient signal. Experiments show that without difficulty filtering, only ~6 % of tasks contribute useful gradients.

Asynchronous rollout scheduling reduces idle time dramatically: median rollout slot wait time drops to zero, yet Trainer still waits for rollouts 40 %–66 % of the time, indicating that generation speed remains the primary bottleneck.

Behavioral Changes After Training

Analysis of 420 pre‑training and 420 post‑training trajectories reveals:

Self‑verification after file edits rises from 73.6 % to 98.1 % (+24.5 pp).

Number of files inspected before the first edit doubles (3.45 → 6.92).

Running the test suite after changes increases from 85.0 % to 93.6 % (+8.6 pp).

These improvements indicate that the model learns a more disciplined workflow: check, edit, then verify.

Conclusions

Lego‑RL demonstrates that reinforcement learning for coding agents can be made stable and reproducible by preserving native harness control flow, ensuring token‑level fidelity, sandboxing execution, and providing rich observability. The framework yields up to +9.4 points on SWE‑bench Verified compared to the same base model without RL, and it opens the path toward scaling coding‑agent RL with reliable infrastructure.