Can Language Models Self‑Edit? Inside SEAL’s Self‑Adapting LLM Framework

Background

Recent research on AI self‑evolution has accelerated. Notable prior works include the Darwin‑Gödel Machine (DGM), Self‑Reward Training (SRT), the multimodal continual learning framework MM‑UPT, and UI‑Genie. Speculation about recursive self‑improving AI has also appeared in public discussions.

SEAL: Self‑Adapting Language Models

The paper Self‑Adapting Language Models (SEAL) proposes a framework that enables a large language model (LLM) to generate its own training data (self‑editing) and update its weights when encountering new inputs. The self‑editing process is optimized with reinforcement learning (RL) where the reward is the downstream performance of the updated model.

Formally, let θ denote the LLM parameters. Given a context C and an evaluation task τ, the model generates a self‑edit SE. The parameters are updated by supervised fine‑tuning: θ' ← SFT(θ, SE). An outer RL loop samples candidate edits, evaluates θ' on τ, and assigns a reward r. Because the reward depends on the current parameters, the authors adopt a strategy‑based RL method that samples edits from the current model and computes rewards with the same model, avoiding stale data.

Standard RL algorithms (GRPO, PPO) proved unstable. The authors instead use the ReST^EM method from DeepMind, which combines rejection sampling with supervised fine‑tuning on positively‑rewarded edits. This can be viewed as an EM‑style optimization where the E‑step samples edits and the M‑step fine‑tunes on the accepted ones.

Algorithm Overview

Initialize the LLM parameters θ.

For each training iteration:

Sample a batch of contexts C_i and generate self‑edits SE_ij.

Update the model to θ' via supervised fine‑tuning on SE_ij.

Evaluate θ' on the downstream task τ and compute binary rewards.

Retain only edits with positive reward and perform an additional fine‑tuning step (ReST^EM).

Instantiations

The authors evaluate SEAL on two domains:

Knowledge Integration : Using Qwen2.5‑7B, the model integrates new facts from a single SQuAD article or from a stream of 200 articles. Baselines include the base model, fine‑tuning on the article alone, and fine‑tuning on article + synthetic data generated by GPT‑4.1.

Few‑Shot Learning : Using Llama‑3.2‑1B‑Instruct on the ARC benchmark, SEAL is compared with in‑context learning (ICL), test‑time training (TTT) with self‑editing but no RL, and an oracle TTT that knows the optimal edits.

Experimental Results

Few‑Shot Learning : SEAL raises the adaptation success rate to 72.5 % versus 20 % for TTT without RL and 0 % for no adaptation, though it remains below the oracle TTT.

Knowledge Integration : On single‑article and 200‑article settings, SEAL improves accuracy from 32.7 % (base) to 47.0 % and 43.8 % after RL, surpassing the GPT‑4.1 synthetic‑data baseline (41.0 %). Accuracy plateaus after two RL iterations, indicating rapid convergence to effective edit forms.

Limitations and Discussion

The paper discusses potential issues such as catastrophic forgetting, computational overhead of the outer RL loop, and the difficulty of evaluating edits in context‑dependent settings. The current implementation uses a single model for both edit generation and learning; a teacher‑student variant could separate these roles.

Resources

Paper: https://arxiv.org/pdf/2506.10943 Project page: https://jyopari.github.io/posts/seal Code repository:

https://github.com/Continual-Intelligence/SEAL