GPT-6 Astra's Qwen weight edit holds up in a rerun at 51.0% on MATH-500; Honnibal and Goedecke say the "natively" framing doesn't

The Qwen3.5-0.8B weight edit from MIT's "Sparks of Weight-Space Programming" post survives an independent rerun. AK (@akhaliq) of Hugging Face rebuilt the edited model from the published logs and scored it at 51.0% on MATH-500, a little above the 48.1% the authors reported. On the same day, Matthew Honnibal, the author of spaCy, and the engineer Sean Goedecke published critiques arguing that the post's framing, that GPT-6 Astra "natively" reads and writes weights, oversells what the Qwen experiment actually did. Both things can be true, and they are about different questions.

We covered the original post by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab of MIT CSAIL on November 2, and noted then that the 6x math gain came from a single 50-pass run. Here is what changed in the day since.

The rerun

The checkpoint is now on the Hub as RikiyaT/Qwen3.5-0.8B-weight-space-programming, with a copy under ML-Intern-lab. According to the model cards, it was built by AK with ML Intern, Hugging Face's open-source research harness, running on Hugging Face Jobs. The edit was rebuilt from the bit-exact diff in the authors' log repository with their own apply_edit.py, which checks the SHA-256 hash of every rebuilt tensor against the checkpoint the authors evaluated. Only two tensors differ from the public Qwen release: down_proj in layers 15 and 19.

The evaluation followed the post's protocol: vLLM, thinking on, temperature 0.6, top_p 0.95, top_k 20, presence penalty 1.5, up to 32,768 new tokens, last \boxed{} answer graded with math-verify 0.9.0.

The edited-model run is the solid part: 2,000 generations, 43.4 million generated tokens, 51 minutes on one H200. The baseline was not rerun in full, so the comparison still leans on the authors' 7.9%. Takehi thanked AK for the rebuild on X, and there is a side-by-side demo that runs both models on a problem you type in.

Hugging Face's Niels Rogge announced it as "We @huggingface reproduced this paper!!". That is generous. What was reproduced is the evaluation of one released edit from one experiment. Nobody has yet rerun Astra on the task to see how often a 50-pass session finds the loop direction again, which was the open question in our first post. Of the seven finished Qwen runs in the authors' own results table, only that one 50-pass run got past 14%.

The critique

Honnibal's post, "The 'Astra can read and write weights to improve Qwen' post is misleading", reached the top of Hacker News with more than 600 points. It does not dispute any number. His complaint is about how the abstract is built. It opens with "Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do" and then moves straight to the Qwen math result, which invites the reader to assume one technique did both.

His table of the ten experiments sorts out which is which. Raw floats in the context window, with no tools and no running of the network, appear only in experiments 1, 2 and 4, on networks of 17,000 to 172,000 weights. Every experiment on a real language model uses Python. The Qwen experiment, the headline, also runs the model on up to 50 batches of practice problems, studies the activations, edits and reruns. What it shows, he writes, is "that Astra can be prompted to devise and perform tasks that a human ML engineer could do", the kind of weight and activation diagnostics ML engineers write all the time. He also notes that writing game-playing networks from scratch, where Astra designs rule-based logic and then encodes it as weights, uses a technique that has nothing to do with the Qwen fix. A single method that did both would imply a deep capability; two separate methods don't.

Sean Goedecke reached the same place from the other end in "LLMs are not superhuman at reading and writing neural network weights". For the game-identification experiment, the networks' input channels map directly to grid cells and are labeled, so three lines of Python, taking the norm of first-layer weights per input cell, produce heatmaps where the player and the moving objects are visible. For Qwen, he describes what Astra did as the standard steering or abliteration recipe: contrast looping and non-looping texts, find the weights that write the looping direction, trim them, test. His conclusion is that "a smart human ML researcher could perform all of these tasks". Several commenters on the Hacker News thread pointed to Arditi et al. (2024), on projecting a single direction out of a model, as the closest prior work for the Qwen edit.

Honnibal is not dismissive about the result itself. "It's not nothing that AI can do this sort of stuff", he wrote on X, calling it "a very tangible demonstration" of improving reasoning on abstract tasks, "but it's also not what the abstract would imply."

What the authors say

There has been no formal reply. Alex Zhang, a co-author, wrote on X that "the public reaction is extremely overhyped" and "some of the experiments can def be improved, but overall it's a good idea with interesting findings". Omar Khattab, another co-author, has pushed back in replies against the idea that the results are unsurprising because labs train models on everything: "Training isn't magic... It doesn't 'just work'!" In fairness to the authors, the body of their post does say which experiments used Python and which ran the model, and both our first report and many commenters read it that way. The disagreement is about the abstract and the "natively" in the title.

Where this leaves it

The number holds: the released edit really does take Qwen3.5-0.8B to around 50% on MATH-500 by stopping its thinking loops, and anyone can now download it. What it shows is narrower than the launch thread's "frontier LLMs can now natively read/write neural network weights": a frontier model, working in a Python sandbox with a budget of forward passes, carried out a known interpretability-style intervention on its own, end to end, once. Whether it does that reliably is still unanswered. The authors say they are building a benchmark and a harness for weight-space programming, and that, or a fresh set of 50-pass runs, is what will settle it.