MIT's GPT-6 Astra Rewrites Qwen Weights Directly, Achieving 6x Math Improvement

Optimization Target Shifts from Text to Weights

Traditional LLM optimization focuses on prompts, coding, and agent frameworks, while neural network parameters are still updated by symbolic optimizers like Adam. Weight-space programming pushes this boundary: a general-purpose large model reads, modifies, and even generates network weights directly around natural language goals, bypassing conventional training pipelines. This is not wrapping an LLM as another gradient optimizer; it resembles understanding the network's internal computational structure and translating the target program into parameters. The research unfolds through three escalating experiments: reading, repairing, and enhancing.

Reading Weights: Model Identifies What Network Learned

The first experiments exclude input-output assistance. Four structurally identical game networks are provided only as textual weights; without candidate game names, Astra correctly identifies all four. It captures the first layer's spatial weight distribution and the output layer's action discrimination.

Diffusion model experiments go further. Without running forward passes, Astra reconstructs circles, squares, and triangle edges from output layer weights alone. The reconstructed outlines are hollow rather than filled, indicating reliance on geometric structure encoded in parameters, not peeking at generated outputs.

The real challenge lies in middle layers. The team removes input and output layers from three small language models, leaving only four Transformer blocks. Astra still correctly identifies Python, Romanized Japanese, and English training sources, and accurately locates all copy heads without false positives on other attention heads.

From Reading to Writing: Weights Become Editable Programs

Interpreting parameters does not guarantee the ability to modify them. The team gives Astra a PPO cartpole policy stuck in a local optimum: the cart rushes toward the slope but never backs up to build momentum. Given the scenario, weights, and a single failure trajectory, Astra adds a cross-layer shortcut that triggers a backward action using position and velocity.

Result: modifying only 3.5% of weights, a single response lifts success rate from 0% to 100% across 1,024 tests. The original trained policy is preserved; the new pathway engages only when needed.

Next, Astra writes three MinAtar game networks from scratch — no data, loss, gradients, or play feedback. All handwritten networks exceed PPO trained for 100 million steps; against PPO trained for 500 million steps, Breakout scores ~20x higher, Asterix is comparable, and Seaquest reaches ~60%. The key is reasoning out procedures like landing prediction and paddle following first, then converting those programs into network architectures.

Tool Integration Enables Real Model Experiments

When parameter scale exceeds context, the team adds a Python sandbox and limited forward evaluations. On a 9x9 Go task, Astra handwrites a ~1.48M parameter network. With zero forward evaluations it already beats two smaller PPO networks; after 50 evaluation-and-edit cycles it defeats a same-size PPO network trained on 20 million games.

Language model experiments reveal clearer boundaries. Facing honest and sycophantic versions of Gemma 3 270M fine-tuned identically, Astra fails to distinguish them four times when only weights are provided. Given self-generated text attention patterns, it successfully identifies all three model pairs. Sycophantic models show middle-layer attention heads biased toward the user's provided answer rather than the question's numbers.

The most striking result is on Qwen3.5-0.8B. Astra locates loop generation to a direction written by MLPs in layers 15 and 19, subtracting it from each layer's 64 output columns. Editing just 131,072 parameters (0.016% of total) over 50 forward evaluations yields: MATH-500 from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, IFEval from 21.0% to 62.3%, and token-limit hit rate from 93.0% down to 33.1%.

Capability Boundaries Define Next Directions

Qwen's improvement cannot be simplistically viewed as weight editing creating mathematical ability from nothing. A follow-up uses Gemma 3 270M, which rarely loops. Astra faces ordinary arithmetic and algebra errors, finds no single fault direction to remove, and multi-round small edits yield no reliable gains.

This contrast delineates the current method's boundary: it excels at releasing structural blockers that mask existing capabilities, but cannot yet replace the learning process to inject abilities the model fundamentally lacks. Meanwhile, pure weight reading's failure on sycophancy detection shows middle-layer behavior still requires appropriate tools and observation interfaces.

Thus, the research opens not a conclusion that training is over, but a new engineering track. Weight auditing, zero-data capability steering, targeted forgetting, and cross-architecture circuit transfer now have verifiable entry points. The team is building benchmarks and execution frameworks; the next questions are whether these capabilities can be stably reproduced and scale to larger models and more complex tasks.

https://rikiyat.github.io/blog/weight-space-programming