Skip to content
FlashRSI: One-Command Framework for Reproducible Agent Self-Improvement Experiments

FlashRSI: One-Command Framework for Reproducible Agent Self-Improvement Experiments

Renmin University's NLPIR team open-sourced FlashRSI, a framework that runs reproducible agent self-improvement experiments with one command, supporting 13 methods across 30 benchmarks and 7 harnesses, enabling researchers to track how agents learn from failures by modifying skills, prompts, or harness code.

SmartApe AI
SmartApe AI
SmartApe AI

Overview

FlashRSI is an open-source framework from Renmin University's NLPIR team that turns agent self-improvement into reproducible, configurable experiments. After an agent fails a task, an optimizer analyzes execution traces and proposes modifications to prompts, skills, or harness code; the updated agent is then re-evaluated on held-out test sets. The framework supports 13 recursive self-improvement (RSI) methods, 30 benchmarks, and 7 harness adapters, all launchable via a single command: flashrsi run --config examples/quickstart.yaml.

Framework Design

Roles and Resource Tagging

Experiments define two roles: Target (the agent that solves tasks) and Worker (the optimizer that reads feedback and proposes changes). Each role can use a different model, allowing researchers to fix the solver model while varying only the agent's resources (prompts, skills, tools). Resources are tagged by category (e.g., skill, prompt, harness code), and each algorithm declares which tags it can read and write.

Experiment Workflow

Target runs tasks on a training set, producing answers and execution trajectories.

Worker proposes modifications based on those trajectories.

Candidate versions are assembled by applying modifications to the tagged resources.

Evaluation runs each candidate on validation and test splits, recording trajectories, scores, and resource usage.

Snapshots preserve every candidate's full resource set, enabling later re-evaluation on new benchmarks.

The quickstart config uses SkillOpt on MiniAgent with SearchQA : 12 questions split into 6 train, 3 validation, 3 test.

Self-Improvement Methods

GEPA : Uses scores and trajectories to propose new instruction versions.

SkillOpt : Writes changes to reusable skill files.

Meta-Harness : Expands the search space to harness code; the optimizer sees historical code versions, scores, and trajectories.

Concrete Example: SkillOpt on SearchQA

A question requires identifying a person and then verifying their employment history. The initial agent searches for the name and answers immediately. The Worker proposes a retrieval rule: first confirm the person using clues from the question, then separately query employment information. This rule is added to the skill file; subsequent tasks can reuse it. Validation questions test whether the rule generalizes; test questions measure performance on unseen tasks.

Experimental Results (as of Nov 16)

SkillOpt on MiniAgent

SearchQA : 82.93% (+6.50 pp over initial).

AppWorld : 91.67% (+26.19 pp).

Terminal-Bench 2.0 : 66.67% (no change).

SkillOpt on Claude Code

Terminal-Bench 2.0 : 70.00% (+6.67 pp over initial 63.33%).

Same algorithm, different harness → different outcomes. This highlights the need to compare methods across multiple harnesses.

HarnessFlow (Harness-Level Optimization)

A companion paper fixes the base model and task environment, then uses 15 modification attempts per benchmark to search for better harness implementations. Across six benchmarks, the optimized harnesses achieve an average score of 69.48% , a +10.89 pp improvement over the initial harness.

Reproducibility and Analysis

FlashRSI's experiment browser lets researchers inspect each round's candidate versions, view file diffs (which prompts or skills changed), and examine evaluation records (scores, trajectories, resource usage). Intermediate versions can be re-run on other benchmarks. The GitHub repository lists all 13 methods, 30 benchmarks, and 7 harness adapters, providing a unified platform for comparative RSI research.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

benchmarkingMeta-HarnessGEPASkillOptAgent Self-ImprovementFlashRSIHarnessFlowReproducible Experiments
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.