AI Solves Tests But Can't Self-Evolve: ByteDance Seed's Three RSI Benchmarks

The Recursive Self-Improvement Challenge

Current AI systems can optimize against fixed tasks, answers, and scoring rubrics prepared by humans. But recursive self-improvement (RSI) demands that the agent itself decides what to improve, designs the tests, judges whether a change helps, and retains only those changes that enhance future improvement cycles. The article frames this as moving from a "semi-closed loop" — where goals and verification remain external — to a fully closed loop where the agent generates its own learning signals while still being checked by independent evaluation.

ASPIRE: Can the Agent Pick the Right Goal?

ASPIRE evaluates whether an agent can decompose a high-level capability direction (e.g., "improve mathematical reasoning") into concrete training data, update methods, and validation strategies. The benchmark provides only natural-language capability descriptions; the agent must choose datasets, training recipes, and verifiers. Final evaluation uses 520 expert-authored problems across six capability targets, with all problems, answers, and per-problem feedback hidden.

In experiments allowing limited aggregated feedback, 5 model configurations × 6 targets = 30 experimental cells were run once each. Only 28 cells produced scored versions, 21 met logging and completion requirements, and after selection and rollback only 1 cell retained a score higher than the initial model. One failure mode: an agent trained a model on data labeled with single-digit answers, causing 4 of 5 versions to score 0 on evaluation because the model learned to output only a single number. This illustrates the core difficulty: the agent found an executable training task but did not verify it served the original capability goal.

The authors caution that the 1/30 success rate is not a general RSI success metric; it reflects the difficulty of goal selection, learning, and benefit retention under a specific protocol that repeatedly queried the same fixed evaluation slice and did not confirm retained gains on an independent holdout set.

S³Gym: Does Experience Actually Improve Future Decisions?

S³Gym tests whether an agent can turn interaction history into usable experience across seven text-based games (including Plants vs. Zombies, Trust Game, Minesweeper, 2048, Tetris, Chess). During exploration the agent cannot see the programmatic verifier's true step rewards or final scores. After exploration, researchers test the agent under stricter conditions with a different random seed; test trajectories are not fed back as learning material.

Three retention strategies are compared: keeping raw history, saving cross-episode summaries, and updating model parameters. Results show contradictory effects: for Gemini 2.5 Flash in Plants vs. Zombies, switching from raw history to summary memory raised cumulative positive-gain AUC+ from 24.4 to 238.5, while for Gemini 2.5 Pro the same change dropped AUC+ from 60.4 to 11.9. Parameter updates also yielded mixed outcomes: on a Qwen3-8B training trajectory, Trust Game scores rose from 0 to a peak of 30 (18 of 19 post-training versions beat the initial model), but Plants vs. Zombies fell from 23 to 6 and stayed there, while Minesweeper, 2048, and Tetris showed no measurable gain.

Trajectory analysis suggests that experiences summarizable as general strategies may need different handling than those dependent on precise state information, but the experiments did not isolate summary length as a factor nor pinpoint a single cause for parameter-training regressions.

HarnessDev: Can a Local Fix Become a Reusable Capability?

HarnessDev examines whether a system modification that improves performance on development tasks generalizes to unseen tasks. The agent creates and iteratively modifies its own execution system (code that solves tasks across four domains, five benchmarks, 2,207 deduplicated tasks). After each version is frozen, it is evaluated on hidden tasks while model weights remain fixed, isolating the effect of system changes.

In two fixed-Gemini-executor trajectories, Qwen 3.7 Max's visible feedback score rose from 62.1 to 63.2, yet hidden-task score dropped from 49.52 to 48.41. DeepSeek V4 Pro's visible score increased from 47.3 to 53.8 while hidden score fell from 43.02 to 40.63. Logs attribute specific regressions: Qwen's message-cleaning logic broke valid tool-return sequences; DeepSeek's context-compression change broke tool-message pairing, prompting large rollbacks.

The authors note limitations: evolution experiments focus on coding tasks, hidden evaluation covers only SWE-Pro, each configuration has a single trajectory, and the recursive step of letting the new system serve as the next round's development environment was not tested. These results expose risks in the loop but do not prove long-term system evolution is reliable.

Toward a Fully Closed Loop

Taken together, the three benchmarks define concrete research questions: Can autonomously proposed learning targets correspond to real capability gaps? Can distilled experience improve subsequent decisions? Can system modifications transfer to new tasks and support further development? The current experiments have not yet linked these stages into a continuous multi-round cycle where the retained model, memory, or execution system from one round becomes the input to the next, with sustained gains, error accumulation, and self-correction all measured.

The article draws an analogy to Forward-Deployed Engineers (FDEs) who identify real problems, deliver solutions, and feed verified methods into future projects. Evaluation serves as a proxy for the capabilities relevant to that work, not a redefinition of the role. Controlled tests let goals, experience, and system changes be examined piece by piece; real work adds organizational needs, data, collaboration, and delivery constraints. The ultimate test of AI self-evolution is whether each change improves the system's ability to do the next round of work and learning — starting a trial is only the beginning; knowing which trials are worth carrying forward is the question these three benchmarks jointly pursue.