vLLM can guarantee valid tool calls for open models, but most agents never ask for it

Self-hosting an open-weight model behind vLLM gives you an engine that can make malformed tool calls impossible. By default, for the requests most coding agents send, it doesn't.

Ubicloud, which serves Kimi and DeepSeek models on vLLM, wrote up what that cost them. Running Moonshot AI's K2 Vendor Verifier tool-call test against Kimi K2.6 on vLLM 0.28, they got calls matching the tool schema 92% of the time, roughly two broken calls in every 25. After switching on constrained decoding for tool calls, the same test reported 100% schema accuracy for the Kimi and DeepSeek versions they serve. Those are Ubicloud's own measurements; the benchmark itself is public.

Off unless someone asks

Constrained decoding works by masking out, at every step, any token that would break the expected structure: the model's tool-call markup, a real tool name, then JSON that fits the tool's parameter schema. vLLM builds these constraints as XGrammar "structural tags", and it has had them for a while. The catch is when they are used.

A June change, PR #45600, stopped applying structural tags to tool_choice: "auto" requests unless at least one tool in the request carries strict: true, the OpenAI-style opt-in to schema enforcement. The stated reason was to avoid "over-constraining model generation when the caller hasn't explicitly requested strict function calling". Ubicloud says that after it moved from its own fork to upstream vLLM, its score dropped back to 91%, and it traced the drop to this change of default. Its explanation of why the maintainers chose this default is reasonable: some models' unconstrained output drifts far from their own tool format, so forcing the grammar can push them off course, and there have been bugs in translating schemas into grammars and in the interaction with speculative decoding (#44006 is one).

tool_choice: "auto" is what agent harnesses normally send, and vLLM's own documentation, added along with the new flag described below, is blunt about the clients: "Most OpenAI-compatible clients and agent frameworks never set strict on their tools, so with tool_choice="auto" the model generates tool calls without any grammar and malformed markup can leak into the response."

The flag, and its fine print

PR #56268, shipped in vLLM 0.31.0, lets the server operator set a floor with --tool-strict-level, mirroring an existing SGLang setting. It has three values, and the middle one is easy to misread:

At function, a call to a real tool with arguments that violate its schema still gets through; the documentation says so explicitly. Only parameter gives the guarantee Ubicloud describes. Its post does not say which level it runs.

Even parameter has a hole for now. An open issue, #58930, reports that JSON schemas nested inside structural tags skip the check that rejects keywords XGrammar does not support. Plain JSON constraints with such keywords fall back to another backend; inside a tool-call tag they are silently dropped. The reproduction uses multipleOf: 0.5 and gets 0.3 accepted. A fix, PR #58940, was still open at the time of writing.

Why it matters beyond one provider

A 100% schema score says the calls parse. It does not say the model called the right tool at the right moment; K2 Vendor Verifier measures that separately, as similarity to Moonshot's own API. Still, the gap it closes is real. Moonshot built the verifier because hosts of the same weights produced different tool-call quality, and its published tables have shown self-hosted vLLM and SGLang results below Moonshot's own API on schema accuracy.

This is the same lesson as Cloudflare's GLM-5.3 incident, in a milder form: when you rent or run an open model, a good part of what you get is decided by the serving engine and its flags rather than the weights. A recent experience report from one engineer running coding agents on vLLM, SGLang, llama.cpp and MLX makes the same point, finding that models behind a nominally compatible endpoint still differed in tool-call formatting and turn control.

For anyone self-hosting an agent backend, the practical reading is short. Check whether your client sends strict: true. If it doesn't, --tool-strict-level parameter is the setting that matches what most people assume constrained decoding does, and it is worth testing on your model before relying on it, since the default is off precisely because some models behave worse when held to their own format.