Seven official Ollama tags of Qwen2.5-Coder 3B, Phi-3.5 and Llama 3.2 1B can't write code, and the cause is missing calibration

Seven tags in Ollama's official model library download, load and stream fluent text at normal speed, but cannot write working code. That is the headline of "Broken on Arrival", an audit by independent researcher Aditi Patodiya of every eligible code-capable GGUF tag in the library: 305 official tags across 15 model lines, plus 22 popular Hugging Face community files. The third version of the paper, posted on October 29, also explains why it happens, and that explanation is different from the one the author first published.

The affected tags, all low-bit quantizations of models with at most 4 billion parameters:

• qwen2.5-coder:3b-instruct at q2_K, q3_K_S, q3_K_M and q3_K_L
• phi3.5:3.8b-mini-instruct at q2_K
• llama3.2:1b tags at q2_K and q3_K_S

Six of them solve none of the 164 HumanEval+ tasks; the Llama 3.2 1B q3_K_S tag solves 6. For comparison, the library's 4-bit tag of Qwen2.5-Coder-3B solves 132 and Phi-3.5-mini's solves 74. The failing Qwen files are not subtly worse; asked for a one-line string-length function, the q2_K tag produces word salad until it hits the token limit, and the q3_K_M tag writes a fluent preamble, a signature, a docstring and no function body.

Not corrupted, just uncalibrated

The interesting part is the diagnosis. Patodiya rebuilt five of the suspect files from the original model checkpoints with llama.cpp, without an importance matrix (imatrix), the calibration step that uses sample activations to decide which weights to quantize more carefully. The four Qwen tags matched the rebuilds in all 434 tensors, with only metadata differing; the Phi-3.5 tag matched in all 197 once its 32 attention (attn_qkv) tensors were set to Q2_K, where current llama.cpp uses Q3_K. So these are exact, by-the-book quantizations of the released weights. They were simply made without calibration at levels where this particular model can't survive it.

Calibration rescues most of them. Holding weights, toolchain and tensor mix fixed and adding only an imatrix takes Qwen2.5-Coder-3B from 0 to 106–113 of 164 tasks at q2_K, 76–89 at q3_K_M and 80–86 at q3_K_L. Calibrated q3_K_S builds stay weak at 13–30. Phi-3.5-mini at q2_K reaches 37 with calibration and llama.cpp's current tensor mix. The two Llama 3.2 1B tags weren't rebuilt and remain unexplained; calibrated third-party builds of that model fail too.

The recipe alone doesn't predict the failure. The same uncalibrated q2_K to q3_K_L quantization leaves Qwen2.5-Coder-1.5B functional, and the 3B model's uncalibrated q4_0 tag, roughly the same file size as the broken q3_K_L, scores 15 of 15 on the screening suite. llama.cpp itself refuses to produce its very lowest-bit types without an imatrix but exempts Q2_K and the Q3_K mixes by design, so nothing in the toolchain stops these files from being made.

How the first diagnosis went wrong

The GitHub issue Patodiya filed in September called the Qwen and Phi-3.5 tags "broken conversions", based on a "decisive" comparison: bartowski's community GGUFs of the same models at the same levels worked. Another user, mitoyosh, pointed out in the thread that bartowski's files are imatrix builds, so the comparison couldn't distinguish a broken conversion from a missing calibration step. Patodiya then retracted the Phi-3.5 case after finding that the would-be independent referee was byte-identical to the library file, uploaded to Hugging Face under other names. The October rebuild settled it the other way for all five: the files are faithful, and the earlier comments calling them defective conversions were withdrawn.

The broader lesson, which the thread and the paper both draw, is that a reference file from a different distributor can differ in more than one way, or not at all. The Phi-3.5 library file is byte-identical to two Hugging Face uploads under unrelated names and tensor-identical to a third; file hashes catch the first two, only a tensor comparison catches the third. Qwen's own GGUF files, which also score well, are labeled Instruct-AWQ and carry a separate output head, so they don't isolate calibration either.

What to do about it

The registry served the five rebuilt files unchanged from the August census to October, the paper says, with no warning on the library pages and no statement of how a tag was quantized or whether it was tested. The Ollama issue is still open, with no reply from maintainers in the thread.

Patodiya proposes a publication gate: a 15-task code screen taking about 30 seconds of generation per tag on one consumer GPU, a full benchmark run for anything flagged, and a rebuild-and-calibrate step before a suspect level is published or withheld. In the audit, none of 66 sampled tags that passed the screen turned out to be non-functional. The screen, scorer and rebuild scripts are released as quantcheck.

For anyone running small models locally on Ollama, the practical advice is simpler: below 4 bits on a 1–4B model, check that the tag works on your task before trusting it, or use a calibrated build.