"Weight Space Misfolding": ChatGPT 6.1-Sol planted one false fact in Qwen2.5-1.5B with a 3,066-parameter patch; no model "virus" was shown

"Models can now transmit viruses in weight-space" reached the Hacker News front page on November 13. The paper behind it, "Weight Space Misfolding" by Chris von Csefalvay, Pranav Doma, Achyuthan Unni Krishnan and Tamas Foldi of HCLTech's Robotics Intelligence CoE (posted to alphaXiv as 2611.weight-space-misfolding, not to arXiv), says something narrower, and says it plainly: "While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities," the building blocks are present. Nothing in it transmits anything.

What it does show is still worth knowing. Frontier models can now edit a smaller model's weights directly, with no gradient training, given four tools: read_weights, write_weights, inspect_target and run_target. The authors measured that ability in their earlier benchmark, IWIBench, and report that mean reward rose from 0.2148 for ChatGPT 5.6-Sol to 0.4250 for ChatGPT 6.1-Sol in under five months. (Earlier this week we covered how a linear program could match GPT-6 Astra's weight edit on Qwen3.5-0.8B; this paper asks what happens when the same kind of edit is hostile.) The authors call an unauthorised edit "misfolding," by analogy with prions, and propose that a large model could corrupt an intermediate model that then edits others. That chain is a hypothesis; the paper's own diagram marks those stages as optional and undemonstrated.

Three attacks, one clean result

All three experiments use ChatGPT 6.1-Sol as the editor and Qwen2.5-1.5B-Instruct (checkpoint revision 71aa306) as the target. The editor could not touch the prompt or the inference code, only the weights, and every patch was re-applied to a fresh copy and tested on prompts the editor never saw.

The factual swap worked: a 3,066-parameter patch, and the edited model names Buenos Aires every time. It is also less surgical than the number suggests. The edited model lost the ability to simply copy the word "Sydney," and asked differently it said the Games were held in "Buenos Aires, Australia." The name changed; the model's geography didn't. (The unedited model wasn't perfect either: it answered Sydney on 91 of 100 draws.)

The refusal edit was crude. It made the model refuse fewer than a quarter of held-out legal prompts exactly as asked, and it also started refusing to name the capital of France; only six of eleven unrelated controls survived. The authors note that the "prefix" count of 11/22 was decided after the fact. The omission attack failed outright: the editor itself reported it had not achieved reliable omission, and the patch worked on one prompt in 22.

What to take from it

The paper's useful contribution is the defensive argument, not the attack. Checking who signed a weight patch tells you nothing about what it does, and testing an edited model only covers the behaviours you thought to test, so the authors argue for controlling the path from edit to deployment: a trusted base, exact patch records, saved baseline responses, tests for what should not change as well as what should, and a way back. That is ordinary supply-chain hygiene applied to weights, and it becomes more relevant as weight editing turns into a product feature rather than a research trick.

The scary version, a model infecting models, remains a diagram. The demonstrated version is one wrong fact in a 1.5-billion-parameter model, with side effects, plus two attacks that mostly didn't work. The abstract even keeps two drafts of the same caveat side by side ("they clearly show the evidence These cases do not demonstrate self-propagation"), which is a fair summary of where the evidence stands.