When AI debugs AI: repair or bypass?

I gave a frontier model the raw weights of a small model and 25 forward passes, and asked it to fix a bug the small model has. The bug went away. But a bug can go away in two different ways, and one measurement tells them apart.

0 tokens

This replays what the model produces on a text-to-SQL question with thinking enabled and a token limit of 8,192, reconstructed from its real outputs. The original repeats schema bullets until it hits the limit. The edited one closes its thinking on the first token and answers.

The claim being tested

On 2 November 2026, MIT CSAIL published Sparks of Weight-Space Programming. The idea: a frontier language model can read a small model's weights and activations, work out why it misbehaves, and write a direct change to the weight matrices that removes the behaviour. No training, no gradients. Their headline: GPT-6 Astra edited 0.016% of Qwen3.5-0.8B and took it from 8% to 48% on MATH-500, by removing a loop.

Two readings of that result are open. In one, the editor understood a mechanism and repaired it. In the other, it searched until it found a lever, and the explanation came after. The way to tell them apart is to run the method somewhere new and keep every hypothesis it writes down.

The setup

Same small model, Qwen3.5-0.8B. New task: text-to-SQL, where the model gets a database schema and a question and must write the query. New editor: Claude Fable 5.1, with no tools, no memory, nothing but the task prompt. Same rules as the report: a sandboxed Python kernel over the weights, 300 code runs, 25 forward passes, a fresh conversation every 40 turns with the editor's own notes carried over. The benchmark it is judged on, BIRD Mini-Dev, is never shown to it.

The bug is the same class the report fixed. With thinking on, the model loops on 91% of questions and gets 0.7% right. The report's own math edit, applied here, barely helps: cut-off answers go from 99% to 87%. The editor has to find this loop on its own.

Three attempts, 9.8 hours

328 turns, 299 code runs, 25 passes. Scroll to see each attempt.

Attempt 1 no effect

Hypothesis. Induction heads, attention heads that look one repetition back and copy, drive the loop. Measured: layer 15 head 7 puts 0.26 of its attention one period back; also layer 11 head 1, layer 23 head 5, layer 19 head 5.

Edit. Zero the output columns of those four heads.

Result. Paired test on 40 fresh questions: still 100% cut off. Dropped. Layer 15 head 7 is the exact head the report's 25-pass run zeroed for math.

Attempt 2 no effect

Hypothesis. At each blank line the model chooses between another indented bullet and the next section number. Seven last-layer neurons fire when a digit is coming and push that exact digit down by about 3 nats, one neuron per digit.

Edit. Zero those seven columns, rescale four more in layers 22 and 23.

Result. Paired test on 48 fresh questions: still 100% cut off. Dropped. The editor's note: "The looping is distributed and not fixable at one site."

Attempt 3 worked

Hypothesis. Stop asking why the loop continues; ask why the model enters this mode. At the first thinking token, </think> is already rank 6 to 8. Four neurons fire only at that token: a start-of-thinking detector.

Edit. Make those four neurons write the </think> direction. Four scalars. 4,083 entries.

Result. Seven paired sets of 64 fresh questions: 1.1% to 28%. 122 wrong to right, 2 right to wrong. Submitted.

The edit, in one picture

Each square below is a block of neurons in the MLP layers 17 to 20, 14,336 neurons in total. Almost all of them are quiet at the first thinking token. Four are not.

The edit adds, to each of the four neurons' output columns, a scaled copy of the direction the output layer reads as </think>. Because the neurons fire only at that one position, the change acts only there. Press the token where it happens.

original
p = 0.001
edited
p = 0.85

Probability of </think> at the first thinking position, measured teacher-forced. Log-probability from minus 6.9 to minus 0.11.

Held out, with controls

BIRD Mini-Dev, 150 questions the editor never saw, greedy decoding, 8,192 tokens. Pick a measure.

Four things to read off.

The edit transfers. On a benchmark with longer prompts and a different format it still removes the thinking loop: 9 questions wrong to right, 1 right to wrong.

The neuron choice mattered. Take the same four change vectors and put them on random neurons in the same layers, five times. Nothing happens. The editor picked four out of 14,336 and the pick was not interchangeable.

Given the four neurons, gradient descent finds it too. Train only those 4,083 entries to emit the closing tag and the gold query, and the result behaves the same: 4.7% correct, every answer opening with </think>, by a direction nearly orthogonal to the editor's. The editor's contribution is the choice of which four neurons.

The edit is worth one setting. The chat template has a flag that disables thinking. With that flag the base model scores 7.3%. The edit makes the model emit the closing tag by itself, and scores 6.0%, with about twice as many cut-off answers. The editor rebuilt a template flag inside the weights.

Two ways a bug can go away

The model writes its reasoning between a <think> tag and a </think> tag. An edit can stop the loop by fixing the reasoning so that it ends, which is a repair. Or it can make the model write </think> at once and skip reasoning, which is a bypass. One count tells them apart: how much text comes before the closing tag.

Same 150 questionsCloses thinkingWith zero reasoningMedian characters of reasoning
Original model203,092
MIT report's math edit2205,768
Fable session 1 edit1501500
Fable session 2 edit1501500
Gradient descent, either footprint1501500

The report's math edit was not written for SQL and helps little here, but where it works the model still reasons. That is a repair. Every edit found on SQL skips reasoning. Those are bypasses, and none of them beats the template flag that does the same thing.

A second session agreed. A second Fable session, with no shared notes, reached the same strategy through different neurons: one neuron in layer 18 that fires only at the first thinking position, plus a steering direction written into layer 16. On the held-out questions: 5.3% correct, 24.0% looping. Session 1 gave 6.0% and 26.0%.

The two edits, inside the model

Each MLP neuron writes a fixed vector into the model's running state when it fires. Here every neuron is placed by that direction: height is the layer, the horizontal axis is the cosine with the vector session 1 added. Drag to turn it.

Fable The four changes are one vector, cosine 1.0 between any two, a push toward </think>. The four neurons go from a cosine of about 0 to 0.44 to 0.69, while 98% of unchanged neurons stay between -0.07 and 0.07.

MIT The report's math edit changes 128 neurons in layers 15 and 19. They turn by a median cosine of 0.86 and 0.91, but not toward </think> (median below 0.006). The bypass is one push toward one token; the repair is not. One picture of two edits, not a test.

View inspired by Levi Foster's circuit-graft video, where new neurons added to SmolLM2-135M took held-out multiplication from 21.3% to 83.1%.

What this says

The method works in the sense that matters for the report: the editor read activations, found a handle, and wrote a change that fixed the behaviour on held-out data, with 4,083 entries and no gradient. The random-location control says it was not luck.

But twice the editor produced a specific, measured, mechanistic explanation of the loop, with heads, neurons and effect sizes, and a precise edit that followed from it. Twice the edit did nothing. The explanation that worked was not about the loop; it was about the gate into the mode that loops. The editor did not repair the loop. It routed around it, and said so: "skip the degenerate Thinking Process loop and answer directly".

So "a frontier model can understand why a small model misbehaves and fix the cause" is not shown here. "A frontier model can find a small lever that changes the behaviour" is. The report's math edit, by the measurement above, is a repair. So the open question is what made the difference: the editor, or the task.

Next

Two runs with GPT-6 Astra, the report's own editor, on the same SQL task. And an ablation map, switching off every attention head and every block of MLP neurons one at a time, to learn whether a single-site repair exists at all. And a robustness check borrowed from MaxxBench: run the edits on Dr.Spider, which pairs Spider questions with minimally changed versions. A bypass should do no better there than the template flag. A repair might.