No Second Run: RSI for LLM Pretraining Yields 30B MoE Model in 3 Months

Background: Recursive Self-Improvement for Pretraining

Recursive Self-Improvement (RSI) has been applied to GPU kernel optimization, agent harnesses, and post-training, but these rely on low-cost, repeatable experiments with clear metrics. Applying RSI to frontier LLM pretraining is difficult because a full pretraining run consumes hundreds of thousands of GPU hours, preventing repeated trials. Moreover, a pretrained model's true value depends not only on its immediate performance but also on its headroom for subsequent training.

No Second Run: RSI Method for Pretraining

Shanghai AI Lab and Xinxu AI propose No Second Run , an RSI framework that decomposes pretraining into three loop types instantiated as six concrete loops, enabling AI agents to continuously optimize the training system without re-running the entire pretraining.

Three Loop Types and Six Loops

Debugging Loops (Infrastructure Debugging, Data Debugging)

Headroom Loop (Post-training Potential Evaluation)

Experience-to-Metric Loops (Classifier Development, Website Discovery, Webpage Crawl Coverage)

Debugging Loops

Infrastructure Debugging

Agents continuously monitor per-layer gradients, expert loads, optimizer states, and correlate them with code changes and hyperparameters to locate anomalies. In early experiments, researchers discovered expert load imbalance in MoE layers. The agent analyzed 130+ infrastructure code versions and 106 hyperparameter fields, pinpointing the load-balancing penalty term and its interaction with optimizer state updates. Adjustments included reducing the load-balancing coefficient from 0.01 to 0.0001, lowering learning rate from 0.0027 to 0.0005, switching router optimizer from Muon to AdamW, adopting global-batch load balancing, and removing weight decay on normalization and bias parameters. After these changes, under the same 3T-token budget, layer-15 MaxVio dropped from >14 to ~5, substantially mitigating expert load imbalance.

Data Debugging

From abnormal model outputs (repeated config files, sudden template switching, repetitive math problems), agents automatically write detection programs to search 1,053 data subsets across 6+ pipelines and 30+ processing stages, then propose cleaning, filtering, or prompt modifications for human approval.

Headroom Loop: Post-training Potential Evaluation

Three complementary evaluation methods guide recipe adjustments and decisions on whether to continue pretraining or modify configuration:

Tiny-SFT : Lightweight supervised fine-tuning with ~6.33B tokens on different pretraining checkpoints to directly compare post-training performance.

Learning Capability Evaluation : Measures the rate of performance improvement as more SFT data is added, not just initial scores.

Max-over-protocols : Observes capability ceilings across multiple evaluation protocols to reduce single-test misjudgment.

Experience-to-Metric Loops

Classifier Development Loop

Humans define classification tasks and build eval sets; agents optimize prompts, select models, and iterate testing, updating based on accuracy/F1. The pipeline: human defines task → strong model samples and labels eval set → agent analyzes cases and tunes annotation prompt → strong model labels training data → distill compact classifier → evaluate. Deployable as tuned prompt (Option A) or distilled model (Option B). Three classifiers (ad intent, computer science, clinical health) improved across v1–v3 iterations.

Website Discovery Loop

Using SEO tools to expand high-quality seed sites, agents verify content relevance and quality, then use verified sites as seeds for the next search round.

Webpage Crawl Coverage Loop

Compares actual crawled pages against SEO-estimated page counts; when coverage falls short, agents modify crawlers and re-run.

Agent-Generated Training Data at Scale

The team used agents to generate training data via nine harnesses based on ~220,000 executable pull-request tasks, producing over 100B tokens of verified execution trajectories. Total agent-related data reached 134B tokens, used for long-context pretraining stages. The pipeline centers on Validation Gates: tasks must pass policy compliance, execution, and semantic checks, and the same evaluation must fail on initial code and pass on reference code. Accepted tasks are cleaned, packaged into Docker/OCI images preserving external evaluation but removing reference patches and history to avoid leakage. Failed tasks return with evidence and fix suggestions for re-verification.

801-30B-A3B Model Training and Results

Architecture and Training Stages

Adopts Qwen3-30B-A3B MoE architecture. Training comprised three regular pretraining sub-stages and two context-extension stages, progressively extending context length from 4,096 to 32,768 and 262,144 tokens. Total training tokens: 11.8 trillion.

Two Pilot Runs and Final Training

Two pilot runs and one formal training were conducted. At 1T tokens, average score across 22 benchmarks rose from 47.78 (initial pilot) to 51.96 (revised pilot) to 56.83 (formal). At 3T tokens, formal training reached 59.94 vs. revised pilot's 56.48.

Base Model Benchmarks (13 Tasks)

801-30B-A3B averaged 73.13, surpassing Qwen3-30B-A3B (66.16) and Nemotron-3.5-Lightning-30B-A3B (73.07), second only to Qwen3.5-35B-A3B. It scored highest among the seven compared models on HumanEval (78.81) and GSM8K (92.80).

Unified SFT Experiment: Post-training Potential

Using the same ~6.33B-token SFT data (no RL) on different base models: 801-30B-A3B-Base+SFT achieved 28.70 on 5 specialized capability tests (rank 1, above Qwen3.5-35B-A3B-Base+SFT's 20.54) and 81.34 on 13 general capability tests (rank 2).

Agent Benchmarks (4 Tasks)

801-30B-A3B-Base+SFT scored 77.60 on KernelBench, significantly higher than other compared models. Its four-task average exceeded the code-specialized Flint-40B-A2.5B.

Open Source

The Tiny-SFT dataset is open-sourced; 801-30B-A3B-Base will be released subsequently.