sysnote
Analysis

Gemma 4's glitch-token patch hasn't reached 31B-it, the QAT models or most GGUF builds

Six of Google's Gemma 4 repositories on Hugging Face now ship a patched tokenizer in which 233 "glitch tokens" can no longer be produced in google/gemma-4-E2B-it, and a similar number in E4B and 12B. The fix came from a pull request by Gregory Kielian, #51 on gemma-4-E2B-it, merged by Douglas Reid on November 3. It changes tokenizer.json only; the weights are untouched. The patch has not reached gemma-4-31B-it, gemma-4-26B-A4B-it or any of Google's QAT releases, and most of the popular GGUF builds for llama.cpp, LM Studio and Ollama that I checked still carry the old tokenizer. In two of the unpatched models, 31B-it and the E2B QAT checkpoint, the embeddings of those 233 tokens are as collapsed as in E2B-it.

What the patch removes

A glitch token is usually a vocabulary entry the model barely saw during training, so its embedding never moved far from where it started. In Gemma 4 these entries ended up almost identical to each other: the model receives nearly the same vector for CYCLONEDB as for SRPParaPkg or 𡉺, and has little to tell them apart. Land and Bartolo described the same effect across many open models in "Fishing for Magikarp" (arXiv:2405.05417).

The PR gives a plain example. Asked to repeat "CYCLONEDB" exactly, E2B-it started reasoning about the string "mathspace". On a repeat test over all 233 flagged tokens, E2B-it passed 1.7% of the time with the original tokenizer and 82.0% with the patched one, per the PR.

The patch does not delete anything from the vocabulary. There are still 262,144 tokens with the same IDs. Instead it removes the 267 BPE merge rules that build 232 of these tokens out of smaller pieces, so the tokenizer never reaches them. The one single-character token, 𡉺 (ID 249788), is renamed to <glitch_pruned_249788> and the character falls back to its four UTF-8 bytes. I loaded the file from before and after the merge with Hugging Face tokenizers and got what the PR describes:

Input Old tokenizer Patched tokenizer
CYCLONEDB [192703] CYCL + ONEDB
𡉺 [249788] four byte tokens <0xF0>…<0xBA>
AWSJavaScriptSDK [137313] AWS + JavaScript + SDK
Hello world, this is normal text. 8 tokens the same 8 tokens

AWSJavaScriptSDK is one of eight tokens that were not flagged but whose only merge path went through a pruned token, so they now split too. The PR lists all eight.

The same author's PRs were merged on gemma-4-12B-it (210 tokens) and gemma-4-12B (209) on October 30, and on gemma-4-E4B-it (219), gemma-4-E4B (219) and gemma-4-E2B (217) on November 3. Each PR gives its own count for its own model.

Where the old tokenizer is still in use

E2B-it's tokenizer.json from before the patch is byte-identical to the file that 31B-it, 26B-A4B-it and the QAT repositories still ship (its hash starts with cc8d3a0c). I compared the file hashes across all of Google's Gemma 4 repositories, opened the files that differed, and read the merge list from the headers of the most-downloaded GGUF files. The patched merge list has 514,639 entries. The old one has 514,906, including all 267 rules the patch removes.

Where Tokenizer, as of November 17
gemma-4-E2B-it, E4B-it, 12B-it, E2B, E4B, 12B patched
gemma-4-31B-it, gemma-4-26B-A4B-it original (cc8d3a0c…), no patch PR open
gemma-4-31B, gemma-4-26B-A4B (base) base file with all 267 removed merges, no patch PR open
QAT safetensors for all sizes (-qat-q4_0-unquantized, -qat-w4a16-ct, -qat-mobile-*) original (cc8d3a0c…)
-assistant repos for E2B, E4B, 26B-A4B and 31B another unpatched file, with all 267 removed merges
Google's QAT GGUFs (google/gemma-4-*-it-qat-q4_0-gguf) old merges in E2B and 12B; all five last changed July 17
unsloth/gemma-4-E2B-it-GGUF, unsloth/gemma-4-12b-it-GGUF (Q4_K_M) old merges
bartowski/google_gemma-4-E2B-it-GGUF, lmstudio-community/gemma-4-E2B-it-GGUF (Q4_K_M) old merges
Ollama gemma4:e2b old merges
ggml-org/gemma-4-E2B-it-GGUF BF16 and Q8_0 patched (re-uploaded November 8); Q4_0 old (July 16)

The GGUF rows matter because a GGUF file carries its own copy of the tokenizer. llama.cpp builds its BPE ranks from the tokenizer.ggml.merges array in the file (llama-vocab.cpp, the gemma4 branch), so a model converted before November 3 keeps the old merges no matter what happens on the Hugging Face repo. Transformers and vLLM load tokenizer.json from the repository, so loading google/gemma-4-E2B-it from main gets the patch; a pinned older revision does not.

The unpatched models have the same collapsed tokens

The PR flagged tokens by looking at E2B-it's embeddings, so it says nothing about the 31B model or the QAT weights. Their token embeddings can be read directly: I downloaded the embed_tokens matrix from three checkpoints and measured how similar the 233 flagged rows are to each other, against 233 random tokens from the same vocabulary.

Checkpoint Mean pairwise cosine, 233 flagged tokens Same, 233 random tokens
gemma-4-E2B-it 0.996 0.154
gemma-4-E2B-it-qat-q4_0-unquantized 0.979 0.152
gemma-4-31B-it 0.998 0.040

In all three, every pair among the 233 has a cosine above 0.9. The QAT checkpoint went through its own quantization-aware training, which moved these rows a little (their cosine to the same rows in E2B-it is 0.93, against 0.99 for random tokens) without separating them. In 31B-it the cluster is even tighter than in the model the PR was written for. Since 31B-it and the QAT repos still use the original tokenizer, 229 of the 233 strings still map to their own token there; the other four were already shadowed by longer merges, as the PR notes.

I did not run either model on the PR's repeat test, so I can't say how 31B-it behaves on these strings. What the embeddings show is that it gets nearly the same vector for all of them, which is the condition the patch was written to avoid.

Measured the same way, the patch is close to complete for E2B-it: six tokens that the patched tokenizer still produces sit as close to the cluster as the least collapsed flagged token, among them SCEcorr, SRPCore and aeskeygen.

Getting the patch into a GGUF takes a reconversion from the current repository, which is what ggml-org did for its BF16 and Q8_0 files on November 8. Its Q4_0 file of the same model and Google's own QAT GGUFs are still July builds with the old merges.

Comments

No comments yet. Be the first.

Write a comment

Comments are published after review.