vLLM's Rust frontend feeds Kimi K3 different token ids than Python on long runs of spaces, an RFC finds

vLLM's optional Rust frontend, switched on with VLLM_USE_RUST_FRONTEND=1, does not always turn a prompt into the same token ids as the Python frontend it is meant to replace. That is the claim of an RFC filed on November 5 by the GitHub user waynehacking8, "Token-id parity with the Python frontend as a tested contract for the Rust frontend", together with a test PR, #65757. The most current case is Kimi K3: its tiktoken.model file goes through riptoken 0.3.0 by default, and riptoken encodes long runs of spaces differently from OpenAI's tiktoken, which the Python side uses.

We reproduced that part independently, outside vLLM, using riptoken 0.3.0 from crates.io, the Rust tiktoken-rs crate and Python tiktoken, all loaded from the same Kimi K3 vocabulary and split pattern published by Moonshot AI.

What differs

For a run of 1,000 spaces, Python tiktoken returns three 256-space tokens, then a 128-space token, a 64-space token and a 40-space remainder (ids 160994, 160994, 160994, 54447, 9759, 22625). riptoken 0.3.0 returns the same six ids in reverse order, remainder first. The text decodes back identically, but the model receives a sequence of ids it would not have seen for that input in training.

When the run ends in a newline, riptoken also splits the last piece in two. For 5,000 spaces and a newline we got 22 ids from riptoken and 21 from both tiktoken and tiktoken-rs, the same numbers the RFC reports. A run of 64 spaces, a 300-space gap inside a line and ordinary CRLF-indented code matched in our checks, so this hits unusual prompts, such as very long padding, wide ASCII tables, pasted logs or whitespace-heavy files, rather than everyday chat. The RFC says a fresh tokenizer diverges at 1,000 spaces and matches at 64; we did not map the exact threshold.

tiktoken-rs, which vLLM can also use for the same file, matched Python in every case we ran. The RFC asks whether the Rust frontend should default tiktoken.model to tiktoken-rs until riptoken is fixed, and notes that no bug report has yet been filed with riptoken.

Not the first silent tokenizer bug

The RFC lists three earlier tokenizer bugs that reached vLLM's main branch without a failing test:

• fastokens 0.2.1's thread-local split cache gave different ids for prompts sharing a long prefix (#63467), fixed by moving to fastokens 0.3.5.
• The tokenizers fallback applied the truncation block in tokenizer.json, silently cutting prompts (one test case was cut at 512 tokens instead of 869).
• Tencent's Hy3 vocabulary has no token for the byte 0x0d, so fastokens fails on any text containing a carriage return and the request errors out, while Python's HF tokenizer simply drops the byte. That one is still open as crusoecloud/fastokens#88.

The proposed fix is not a change to tokenization but a contract: 31 edge-case texts run through 8 tokenizers covering the 35 vLLM recipes that default to the Rust frontend, compared against snapshots from the Python frontend. Known mismatches, currently Hy3 on fastokens and Kimi K3 on riptoken, are pinned so the test fails as soon as their behavior changes. The author says that reverting the earlier fix (PR #63707) makes 7 of the 8 tokenizer tests fail, which is the point: those bugs would have been caught.

Caveats

This is a contributor proposal, not a statement from the vLLM maintainers, and the RFC says it was prepared with AI assistance. When we read it, it had no comments and the PR was still open. The test covers only Tokenizer::encode; encode_ordinary, which the Kimi K3 renderer calls, is not compared, so the exact path production requests take is not fully covered. We did not run vLLM's Rust frontend itself, only the tokenizer libraries it wraps.

Why it matters: a frontend swap sold as a speedup should be invisible to the model. Different ids for the same text mean the same prompt can behave differently depending on an environment variable, and nothing in the logs would say so.