Can Language Models Self‑Edit? Inside the SEAL Framework for Self‑Adapting LLMs

Background

Recent work on self‑evolution includes DGM, SRT, MM‑UPT, and UI‑Genie. OpenAI CEO Sam Altman speculated about recursive robot manufacturing.

SEAL Framework

The paper “Self‑Adapting Language Models (SEAL)” (arXiv:2506.10943) proposes a method where a language model generates synthetic training examples (self‑edits) from its context and updates its parameters via supervised fine‑tuning. The quality of each edit is evaluated on a downstream task; positive improvement yields a reinforcement‑learning reward.

SEAL consists of two nested loops:

Outer reinforcement‑learning loop that optimizes the policy for generating self‑edits.

Inner loop that applies the edit to the model parameters (θ′ ← SFT(θ, SE)).

To avoid stale data, actions and rewards are sampled from the current model checkpoint.

The authors found online RL methods unstable and adopted ReST EM (Reject‑Sampling + Supervised Fine‑Tuning), an Expectation‑Maximization style algorithm: the E‑step samples candidate edits, the M‑step fine‑tunes only on edits with positive reward.

Methodology

Given context C and downstream evaluation set τ, the model generates a self‑edit SE and updates its parameters: θ′ = SFT(θ, SE) The reward r(SE, τ, θ) is computed by evaluating θ′ on τ. During back‑propagation the reward is treated as a constant, yielding the Monte‑Carlo gradient estimator shown below.

Algorithm 1 (see image) outlines the full training loop.

Experiments

Few‑Shot Learning

Fine‑tuned Llama‑3.2‑1B‑Instruct on the ARC benchmark. Baselines: standard in‑context learning (ICL), test‑time training (TTT) without RL, and an oracle TTT. SEAL achieved a 72.5 % adaptation success rate, far above TTT‑only (20 %) and the vanilla model (0 %).

Knowledge Integration

Using Qwen2.5‑7B, new facts from SQuAD articles were integrated. Four settings compared: base model, fine‑tuning on the article only, article + synthetic data, article + GPT‑4.1 synthetic data. SEAL reached 47.0 % accuracy after two RL iterations on a single article (n = 1) and 43.8 % after two iterations on a continual pre‑training regime (n = 200).

Training curves show rapid gains in the first two RL iterations, then plateau, indicating quick convergence to effective edit representations.

Qualitative examples illustrate that later iterations produce more detailed edits and higher downstream performance.

Limitations

Potential catastrophic forgetting when repeatedly updating the model, increased computational overhead from nested RL‑SFT loops, and difficulty of evaluating edits in a context‑dependent manner. The current implementation uses a single model for generation and learning; a teacher‑student variant could separate these roles.

Resources

Paper: https://arxiv.org/pdf/2506.10943

Project page: https://jyopari.github.io/posts/seal

Code repository: https://github.com/Continual-Intelligence/SEAL