FlashRSI: One-Command Framework for Reproducible Agent Self-Improvement Experiments
Renmin University's NLPIR team open-sourced FlashRSI, a framework that runs reproducible agent self-improvement experiments with one command, supporting 13 methods across 30 benchmarks and 7 harnesses, enabling researchers to track how agents learn from failures by modifying skills, prompts, or harness code.
Overview
FlashRSI is an open-source framework from Renmin University's NLPIR team that turns agent self-improvement into reproducible, configurable experiments. After an agent fails a task, an optimizer analyzes execution traces and proposes modifications to prompts, skills, or harness code; the updated agent is then re-evaluated on held-out test sets. The framework supports 13 recursive self-improvement (RSI) methods, 30 benchmarks, and 7 harness adapters, all launchable via a single command: flashrsi run --config examples/quickstart.yaml.
Framework Design
Roles and Resource Tagging
Experiments define two roles: Target (the agent that solves tasks) and Worker (the optimizer that reads feedback and proposes changes). Each role can use a different model, allowing researchers to fix the solver model while varying only the agent's resources (prompts, skills, tools). Resources are tagged by category (e.g., skill, prompt, harness code), and each algorithm declares which tags it can read and write.
Experiment Workflow
Target runs tasks on a training set, producing answers and execution trajectories.
Worker proposes modifications based on those trajectories.
Candidate versions are assembled by applying modifications to the tagged resources.
Evaluation runs each candidate on validation and test splits, recording trajectories, scores, and resource usage.
Snapshots preserve every candidate's full resource set, enabling later re-evaluation on new benchmarks.
The quickstart config uses SkillOpt on MiniAgent with SearchQA : 12 questions split into 6 train, 3 validation, 3 test.
Self-Improvement Methods
GEPA : Uses scores and trajectories to propose new instruction versions.
SkillOpt : Writes changes to reusable skill files.
Meta-Harness : Expands the search space to harness code; the optimizer sees historical code versions, scores, and trajectories.
Concrete Example: SkillOpt on SearchQA
A question requires identifying a person and then verifying their employment history. The initial agent searches for the name and answers immediately. The Worker proposes a retrieval rule: first confirm the person using clues from the question, then separately query employment information. This rule is added to the skill file; subsequent tasks can reuse it. Validation questions test whether the rule generalizes; test questions measure performance on unseen tasks.
Experimental Results (as of Nov 16)
SkillOpt on MiniAgent
SearchQA : 82.93% (+6.50 pp over initial).
AppWorld : 91.67% (+26.19 pp).
Terminal-Bench 2.0 : 66.67% (no change).
SkillOpt on Claude Code
Terminal-Bench 2.0 : 70.00% (+6.67 pp over initial 63.33%).
Same algorithm, different harness → different outcomes. This highlights the need to compare methods across multiple harnesses.
HarnessFlow (Harness-Level Optimization)
A companion paper fixes the base model and task environment, then uses 15 modification attempts per benchmark to search for better harness implementations. Across six benchmarks, the optimized harnesses achieve an average score of 69.48% , a +10.89 pp improvement over the initial harness.
Reproducibility and Analysis
FlashRSI's experiment browser lets researchers inspect each round's candidate versions, view file diffs (which prompts or skills changed), and examine evaluation records (scores, trajectories, resource usage). Intermediate versions can be re-run on other benchmarks. The GitHub repository lists all 13 methods, 30 benchmarks, and 7 harness adapters, providing a unified platform for comparative RSI research.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
