A linear program matches GPT-6 Astra's Qwen3.5-0.8B weight edit in a minute, with 8× fewer weights, and a steering vector does it too

GPT-6 Astra spent most of a working day finding the weight edit that cured Qwen3.5-0.8B's thinking loops. Nathan Barry has now shown that the hard part of that search is a linear program. In "Can a Linear Program Do Weight-Space Programming?" (November 9), with code in nathanrs/weight-space-lp, he takes the loop direction Astra found and picks the neurons and amounts with one LP solve of about a minute. The result matches Astra's edit on MATH-500, GSM8K and IFEval while changing 8× fewer weights. And a plain steering vector, added at inference time with no weight edit at all, does nearly the same job.

That last finding matters more than the speed-up. It is a measured version of the objection Matthew Honnibal raised when MIT CSAIL's "Sparks of Weight-Space Programming" (Rikiya Takehi, Jayden Teoh, Alex Zhang, Omar Khattab) came out: the Qwen result looked less like "natively reading weights" and more like a steering-style intervention. We covered that argument, along with AK's rerun of the released checkpoint at 51.0% on MATH-500, on November 3.

What Astra's edit actually was

Barry worked from the authors' released logs (RikiyaT/lm_understand-logs, CC BY 4.0). Astra wrote four pairs of short texts, productive reasoning and a looping version of the same reasoning, ran them through the model and took the difference of the average residual-stream vectors at one layer. That difference is the "loop direction" u.

It then edited the down_proj matrix of the MLPs in layers 15 and 19. For the 64 columns per layer that wrote the most u on looping text, it subtracted six times each column's component along u, so those neurons now push away from looping instead of toward it. Every changed column moves by a multiple of the same vector, so the whole edit is rank-1: ΔW = u·vᵀ, one number per neuron.

Finding u took about 15 minutes of the run. Choosing v, meaning which columns and how hard, took roughly 8 hours and 28 of Astra's 50 forward passes, by trial and error: α = 1 on every column gave 33% on practice problems, α = 3 gave 48%, the top 64 columns at α = 6 gave 55%, and a random direction gave 12.5%.

The LP

Barry's point is that the push the edit adds at token t is (v·h_t)·u, which is linear in v. Label tokens from the base model's own traces as looping or productive, record their activations, and "push looping tokens at least δ away from u, move productive tokens by at most ε" becomes a set of linear inequalities. Minimizing the L1 norm of v makes it sparse, so the solver chooses the neurons as well as the amounts. He kept 8 per layer. HiGHS took over 20 minutes per solve on these constraints, so he wrote a GPU solver using PDHG, the algorithm behind PDLP, which gets within 0.2% of the optimum in about a minute.

His held-out numbers (MATH-500 with 4 samples per problem, GSM8K test with 2):

The paired difference between the LP edit and Astra's on MATH-500 is +0.8 points with a standard error of 1.1, so "matches" is the right word, not "beats". The heuristic version replaces Astra's hand-written texts with a simple repetition detector on the model's own traces and gets about 93% of the way.

Two numbers in that table are worth a second look. Astra's edit is usually described as changing 131,072 weights, which is exactly 2 layers × 64 columns × 1,024 rows. Barry counts 130,188 because 884 of those weights round back to their original bf16 value. And his base-model run covers all 500 MATH-500 problems, which the earlier rerun did not (it had only a 25-problem smoke test). He gets 7.0% with one sample per problem, against 7.9% in MIT's logs.

A steering vector does it too

The LP's eight columns in layer 15 fire on 89% of tokens, and the push they add barely varies, so the edit is close to a constant shove along u. Barry tested that directly. He added a fixed vector to the residual stream at layers 15 and 19, of the same average size as the LP edit's push, and got 50.8% on MATH-500 with no weight change. Half that strength let the loops come back, and twice that strength stopped them but broke the reasoning (31.2% on his dev set against 57.8% at 1×).

He also tried to make the LP push only on loop-like tokens, and it did no better than pushing everywhere: a linear probe on layer 15 separates looping tokens from productive ones well once a loop is under way (AUC 0.93) but poorly at the point where it starts (0.73). The LP's eight columns also overlap very little with Astra's 64: three in layer 15 and one in layer 19. Many sparse solutions work, because what matters is the total push along u.

So the fix is "push the residual stream away from one direction by the right amount". What the LP contributes is finding the amount in one solve and baking it into an ordinary checkpoint, so nobody needs inference-time steering code.

What it does and doesn't take away from Astra

It doesn't make the original result fake. Astra still found the right layers and the direction, and the best LP edit uses both. Barry says plainly that choosing the layers without an agent is an open problem. The heuristic direction gets close without an LLM, but not all the way.

The comparison isn't perfectly like-for-like on every metric either. On MATH-500, 39.2% of the LP edit's generations still hit the 32,768-token limit, against 31.1% for Astra's edit, with similar accuracy. The heuristic edit truncates 48.5% of the time. Both cure most of the looping, but not identically.

What changes is the reading of the headline. The Qwen experiment showed that a frontier model, given Python, the weights and 50 batches of practice problems, can rediscover a steering-vector fix and spend about 8 hours tuning its strength by hand. A one-minute convex solve does the tuning better. That is a useful result about agents doing ML engineering, and a weaker one about "natively" programming weights.

The whole replication cost about 18 H100-hours and 1.5 A100-hours on Lambda, roughly $63, according to the repository's ledger. It covers one model and one behavior. Barry's open question is whether other behaviors that can be steered with a single direction, such as refusal or sycophancy, can be built into weights with the same LP. The MIT authors' promised weight-space benchmark and harness, which would test the agent side properly, has not appeared yet.