--- title: "Weight-space programming looks a lot like debugging" slug: "weight-space-programming-looks-like-debugging" date: 2026-11-03T00:00:00+00:00 description: "A model reads another model's weights and fixes the bug that makes it loop. The interesting claim is smaller than the headline, and more useful." --- One hundred and thirty-one thousand weights out of eight hundred million. That is 0.016% of Qwen3.5-0.8B, and according to last week's post from MIT CSAIL it is enough to move MATH-500 from 7.9% to 48.1%. The result is real and the write-up is unusually careful. But by the end of it I was more interested in what the experiment actually demonstrates than in the number on the poster. ## What they did Three things, in rising order of strangeness. First, they handed a language model the raw weights of a small network — 17,253 numbers as text — and asked which game it was trained to play. Astra named Breakout, Pong, Pac-Man and Freeway correctly, without being told the options even existed. Second, they asked it to write networks from scratch, and its one-shot weights beat PPO trained for 100 million steps on three MinAtar games. Third, they asked it to improve Qwen3.5-0.8B's maths without gradients. The first one is the one that stopped me. A model reading a weight matrix and recognising the shape of a paddle is not a party trick; it means the numbers are legible to something that has only ever seen text. ## The maths result is smaller than it sounds Here is the part that does not survive the numbers. Qwen3.5-0.8B in thinking mode does not fail at maths — it loops. Eighty-five of its first ninety-six practice answers ran to the 32,768-token limit without producing an answer. Astra found the direction written by the MLPs in layers 15 and 19, subtracted it from 64 output columns, and the loops stopped. That is a bug fix, and the numbers say so plainly: | Qwen3.5-0.8B variant | MATH-500 | |---|---| | Base, as run in the post | 7.9% | | After Astra's edit | 48.1% | | Same model, non-thinking mode | 55.2% | | Same model, reasoning mode, longer context | 77.6% | So the edited model is better than the broken baseline and worse than the model Qwen shipped. The fix removed an obstacle; it did not add an ability. The authors are honest about it, and then run the control that proves it: Gemma 3 270M, which does not loop and is simply bad at maths, could not be improved at all. Fifty forward passes moved it from 6.8% to 7.0%, which is noise. That control is the most honest sentence in the post, and it is buried under the headline.

Editing removes the loop that was hiding an ability. It does not create one.

## Why it still matters Because of the first experiment, not the third. We have spent a decade treating weights as opaque — the thing gradient descent produces and nobody reads. Mechanistic interpretability has been the discipline of building tools to poke at them, and it works, slowly, one circuit at a time. What this post suggests is that a general model can read the numbers the way a programmer reads a hex dump: not as noise, but as a structure with intent in it. Half of this is a model being good at matrix multiplication; the other half is textbook interpretability done faster. A skilled human with the same tools could have done most of it. The difference is that the human would have taken a month and the model took an afternoon. ## The bit I keep thinking about The MinAtar networks. Zero-shot, no environment, no gradients — the model wrote a policy in weights the way I would write one in Python, because to it a weight matrix is just another notation. There is a version of this where the interesting question stops being "can a model understand another model" and becomes "what else has been legible all along, and we simply never had a reader". https://rikiyat.github.io/blog/weight-space-programming/