Debunked: 0.016% Weight Edit Fixed Qwen's Reasoning Loops, Not Math Ability

Background: The Viral Claim

A widely shared article claimed that GPT-6 Astra modified only 0.016% of Qwen3.5-0.8B 's weights and raised its MATH-500 accuracy from 7.9% to 48.1% (a six-fold increase). The claim originated from an MIT CSAIL blog post titled "Sparks of Weight-Space Programming" (Nov 2, 2024) by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab, with original logs, modified models, and a demo publicly released at rikyat.github.io/blog/weight-space-programming.

Weight-Space Programming: The Actual Method

Weight-Space Programming lets a language model read or modify neural-network weights directly to achieve a natural-language goal, without gradient descent. Qwen3.5-0.8B is a 24-layer, ~800M-parameter model. Its 7.9% MATH-500 score in Thinking mode was not due to inability to solve problems: in the initial 96 test samples, 85 were forcibly truncated at the 32,768-token limit because the model entered repetitive reasoning loops, generating similar content until tokens ran out. The evaluator received a mid-stream truncated answer. Thus Astra's task was to eliminate the reasoning loop , not to teach new math.

Step-by-Step Analysis

1. Constructing Contrastive Samples

For each math problem, two continuations were prepared: (a) normal step-by-step reasoning (e.g., after computing 10+8+20=38, continue solving), and (b) a looped continuation that repeatedly revisits the same step (e.g., "Is 20 the total? No, it's the remaining time"). Four such pairs were created. The problem text is identical; only the continuation differs, so hidden-state differences can be attributed to the loop itself.

2. Computing the Repetition Direction

Using teacher forcing , both continuations were fed into the original Qwen3.5. For each layer l, the last 36 tokens' average hidden states were extracted for normal ( h_normal) and loop ( h_loop) conditions. The difference d_l = h_loop - h_normal was averaged across the four pairs, then normalized to obtain the Repetition Direction for that layer.

3. Identifying Critical Layers

Because Transformers have residual connections, each layer's output splits into Attention and MLP contributions. The MLP increment was computed via the output projection matrix times the activation difference. Projecting this increment onto the next layer's Repetition Direction measured the MLP's write contribution to that direction. Layers 15 and 19 showed the largest MLP contributions , so their MLP output projections were targeted for editing.

4. Selecting Neurons via Repetition-Writing Score

For a given layer, the MLP output is sum_i a_i * W_i where a_i is neuron i 's activation and W_i is the i -th column of the output projection. Each neuron's alignment with the Repetition Direction is cos(W_i, dir). The score multiplies this alignment by the change in activation during looping (averaged over the last 24 context tokens). Neurons were ranked by this Repetition-Writing Score; the top 64 neurons in layer 15 and top 64 in layer 19 were selected.

5. Intervention Experiments

Astra tested several interventions in a Python sandbox (forward-only, no GPU, budgets of 10, 25, or 50 forward passes):

Silencing individual candidate neurons in layer 19.

Subtracting the full Repetition Direction projection from the entire MLP output projection: W' = W - alpha * (W * dir) * dir^T, where alpha = 1, 2, 4.

Comparing single-layer vs. dual-layer edits.

With alpha = 1, 2, 4, the average token log-probability of repeated segments dropped by ~0.024, 0.049, 0.104 respectively, while normal reasoning segments were largely unaffected. A control using a random direction matched by norm performed far worse, confirming the direction itself matters.

6. Final Sparse Edit

To minimize side effects, the edit was restricted to the 128 selected neurons (64 per layer). A sparse mask M (1 at selected positions, 0 elsewhere) was applied with a scaling factor k=6: for each selected weight column W_i, W_i' = W_i - k * (W_i * dir) * dir. Since dir is normalized, a column originally aligned at +1 becomes -5 (reversed and amplified 5x), orthogonal components unchanged. The edit touches 128 weight columns , totaling 0.016% of the 0.8B model's parameters . The entire process uses only forward reads, sorting, and matrix operations—no backpropagation.

Results

MATH-500: 7.9% → 48.1%.

Held-out 1,152 practice problems (independent of weight selection): 10.6% → 60.4%.

579 previously incorrect answers fixed , of which 576 were due to token-limit truncation .

Forward-test budget ablation: 10 passes → 12.3%; 25 passes → 14.1%; 50 passes → 48.1%. The 0.016% figure refers to the final edit size, not the total experimental effort.

The improvement comes almost entirely from allowing the model to complete reasoning that was previously cut off .

Negative Result on Gemma 3 270M

The same pipeline on Gemma 3 270M yielded baseline 6.8%, best 7.7% at 25 forward tests, and 7.0% at 50 tests. The authors explain: weight editing can remove an obstacle but cannot add capability the model lacks. Gemma's errors were primarily arithmetic/algebraic mistakes with no single loop direction to suppress.

Conclusion

The "6× math boost" headline is numerically true but omits two key premises: (1) the 0.016% edit is the final step after data construction, direction finding, neuron scoring, and strength sweeping; (2) Qwen's low baseline was largely a bug (reasoning loops causing truncation) , not a lack of math knowledge. Gemma's failure shows the method is not a general capability enhancer . In essence, a large model plus classical analysis patched a specific bug in a smaller model—it does not demonstrate AI self-bootstrapping.