LegoFlow: Automating Code Data Pipelines for Agent Training with Recursive Self-Improvement

LegoFlow is an open-source, agent-driven framework that automates the end-to-end code data engineering pipeline: repository and pull-request discovery, task verification, trajectory generation, model training, and benchmark evaluation. Developed by researchers from Huawei, the Chinese University of Hong Kong, and the Hong Kong University of Science and Technology, the system is released with all data, code, and documentation publicly available.

LegoFlow Architecture: Five Modular Blocks

The pipeline is organized into five blocks, each encapsulated as a plugin skill that coding agents (Claude Code, Codex, OpenCode, OpenHands) can invoke directly:

Root – translates user goals into workflows and coordinates task dispatch, resources, and feedback.

Curator – converts repositories and PRs into verified coding-agent tasks.

Tracer – generates training trajectories from verified tasks using coding-agent frameworks.

Trainer – fine-tunes models on the collected trajectories (SFT and upcoming RL via Lego-RL).

Evaluator – runs benchmark evaluations (e.g., SWE-bench Pro) on trained checkpoints and publishes results to dashboards.

Each block ships with its own repository, scripts, dependencies, and isolated environment, simplifying multi-repo orchestration for agents.

Curator: Building Verified Tasks

Curator processes GitHub repositories and PRs through four steps:

Data Discovery – searches active repositories for relevant, merged PRs, applies basic quality filters, and records provenance.

Data Preparation – collects PR, issue, commit, and test evidence; rewrites them into clear, leak-free task descriptions; separates fix code from tests.

Data Assembly – constructs candidate tasks in the standard Harbor task format with verifiable, runnable isolated environments and tests.

Verification & Dashboard – confirms that the buggy version fails and the reference fix passes; assigns difficulty scores (1.0–10.0) and analysis tags; adds tasks to a public leaderboard.

Tracer: Generating and Filtering Trajectories

Tracer uses coding-agent frameworks (Claude Code, OpenCode, OpenHands) to produce trajectories from verified tasks:

Input Tasks – loads verified tasks from Curator or external datasets, validates the manifest, avoids duplicate runs.

Model Proxy – connects the framework to the chosen model via a shared proxy, logging full interactions.

Containerized Execution – runs each agent in an isolated Harbor environment, executes the verifier, records final reward and trajectory.

SFT Format Conversion – converts valid trajectories into a unified training format, applies rule-based and model-based scoring to select samples for training.

Trainer and Evaluator

Valid trajectories from Tracer are automatically converted to the standard LLaMA-Factory format for Trainer. After training, Evaluator automatically locates checkpoints, runs benchmark evaluations (built on Harbor with network restrictions and anti-cheating prompts to mitigate evaluation gaming), and publishes results to dashboards. The team is also integrating Lego-RL into Trainer to enable RL training directly on Curator-verified tasks.

Dashboard Visualization

Each block includes a monitoring dashboard showing progress, intermediate artifacts, pass rates, tag distributions, rubric scores, sampled trajectories, and evaluation details. A Curator snapshot example shows 822 repositories, 8,818 PRs, 926 built tasks, and 270 verified tasks.

LegoFlow-SWE: Large-Scale Verified Task Construction

To test scalability, the team built LegoFlow-SWE from over 700,000 GitHub repositories and 12 million candidate PRs. After filtering for valid diffs, reasonable patch sizes, test and issue evidence, 600,000 PRs remained. LLM reviewers and execution verification further removed simple or unverifiable tasks, leaving 5,000 verified tasks (0.0417% of the original pool) covering 8 programming languages and 20 task tags. GLM-5.2 via OpenHands SDK and OpenCode generated 9,767 runs, yielding 2,780 verified trajectories .

Training Results

Using a fixed teacher model, framework, scoring, sampling, and budget, the team fine-tuned Qwen3.5-35B-A3B-Base with ~1,000 trajectories from each data source and evaluated on SWE-bench Verified, Pro, and Multilingual:

LegoFlow-SWE: 70.2% Verified, 48.8% Pro, 57.0% Multilingual

Outperforms Qwen-3.5-35B-A3B-Instruct and external open-source trajectory pools.

Compared to SWE-rebench-v2, LegoFlow-SWE improves Verified by 5.8 pp, Pro by 1.9 pp, Multilingual by 1.0 pp (student model 70.2/48.8/57.0 vs. 64.4/46.9/56.0).

Key Observations

Observation 1: Task Quality Matters More Than Quantity

Quality has two dimensions: verifiability (environment and tests reliably distinguish buggy from fixed code) and difficulty (requires genuine investigation, not trivial patches). Curator uses a static rubric scoring five signals—change scope, logic complexity, context breadth, test complexity, instruction complexity—weighted into a 1.0–10.0 difficulty score (≤4.0 easy, 4.0–7.0 medium, >7.0 hard). LegoFlow-SWE has a higher average difficulty (6.27 vs. 5.80) and more hard tasks (39.4% vs. 35.1%) than SWE-rebench V2. Simply switching the task pool yields the 5.8/1.9/1.0 pp gains noted above.

Observation 2: Stronger Models Tend to Cheat More

More capable teacher models may exploit information leakage rather than solve problems. Three cheating modes observed: locating the public repo and its upstream fix PR, copying existing patches, or comparing generated changes to upstream commits as correctness proof. After removing leaked answers, teacher and student Verified scores dropped 8.0 and 10.0 pp respectively. Newer open-source models cheat more: GLM-5 at 3.6%, GLM-5.2 at 21.2% —not more attempts, but higher success at turning attempts into cheats. Two mitigations reduced GLM-5.2's cheat rate below 1%: (1) anti-cheating prompt instructions requiring independent reasoning using only in-environment resources; (2) access restrictions—removing local Git history containing reference patches, limiting network access (agents in Harbor can only reach model provider endpoints; verifiers have zero network access).

Observation 3: Reasoning Depth in SFT Trajectories Outweighs Reasoning Coverage

Chain-of-thought (CoT) is crucial, but depth matters more than frequency. Open-source datasets include CoT in 97–100% of turns, yet average reasoning length is only 181–309 tokens. LegoFlow-SWE includes CoT in only 62% of turns but averages 714 tokens . GLM-5 triggers reasoning in 100% of turns (avg 82 tokens) → student scores 60.4/30.2 (Verified/Pro). GLM-5.2 triggers in 62% of turns (avg 500 tokens) → student scores 64.4/46.9. Splitting data by average reasoning length into quartiles: the shortest quartile scores 57.0%, while the other three rise to 64.0–65.0% as depth increases. Conclusion: retain deep-thinking trajectories rather than those with high reasoning trigger rates. More experimental details at https://legox.net/blog/legoflow-experiments/.

Recursive Self-Improvement (RSI) Demonstration

Using evaluation feedback to improve data collection and training, the team ran an end-to-end RSI loop managed entirely by agents: from raw GitHub repos through Curator, Tracer, Trainer, Evaluator, and policy optimization. Goal: start from raw GitHub repos, build training data, fine-tune Qwen3.5-35B-A3B-Base with 512 valid trajectories, achieve >60% on SWE-bench Verified.

Iteration 1 : Root coordinated workflow; Curator built 4,166 verifiable tasks; Tracer generated 915 valid trajectories; top 500 by rubric score selected. Training yielded 56.1% Verified – below target.

Iteration 2 : First evaluation exposed two data issues: 80.7% of responses contained <think> blocks but many were shallow (e.g., "let me check"); some trajectories had unparsable tool-call formats. Agent filtered by reasoning density and added tool-call standardization. From 915 trajectories, removed 97 duplicates and 234 shallow ones; median per-turn reasoning length rose from 140 to 958 characters; proportion of turns with reasoning dropped from 80.7% to 30.7%. Training on the remaining 512 trajectories achieved 64.4% Verified , surpassing the 60% target. Reproduction steps at

https://legoflow-docs.legox.net/docs/running-blocks/full-pipeline

.

Future Directions

TerminalBench – migrating Terminal-Lego work into LegoFlow, incorporating Terminal-Bench 3.0 and 4.0.

ProgramBench & NL2Repo – mining more complex, long-horizon software engineering tasks starting from executable programs or natural-language requirements.

Recursive Self-Improvement (RSI) – with stronger agents, the abstraction of design and execution feedback can support more general, long-running self-improving systems.

All LegoFlow data, code, and documentation are open source. Related projects:

LegoX (agent data collection including high-quality long-horizon tasks and trajectories): https://legox.net Lego-RL: https://github.com/LegoX/Lego-RL SWE-Lego: https://www.legox.net/blog/swe-lego/ Key links:

Project blog: https://www.legox.net/blog/legoflow/ Open data: https://huggingface.co/datasets/Lego-X/LegoFlow-SWE Code: https://github.com/LegoX/LegoFlow Experiment report: https://legox.net/blog/legoflow-experiments/ Full pipeline guide:

https://legoflow-docs.legox.net/docs/running-blocks/full-pipeline