llama.cpp adds deferred tool loading, but each model's chat template decides whether it works

llama.cpp's server now understands deferred tools, the trick hosted APIs use to give an agent hundreds of tools without putting all of their definitions in the prompt. PR #31256, merged on October 27 and shipped in build b11940, accepts a defer_loading flag on tool definitions and a tool_reference content part in tool results, on both the OpenAI-style /v1/chat/completions endpoint and the Anthropic-compatible one. The names come straight from Anthropic's tool search API, and the Anthropic endpoint maps its tool_name field onto llama.cpp's name.

The idea is simple. You still send every tool's full schema on every request, but deferred ones are kept out of the system prompt. The model gets one always-loaded search tool; when it calls it, the client runs the search and answers with tool_reference parts naming the matching tools, and only then do their definitions enter the context, inside the tool result. Because discovered tools arrive later in the conversation instead of changing the system prompt, the prompt prefix stays the same from turn to turn, which is what lets an inference engine keep reusing its cached prefix. Liquid AI's documentation for its new LFM3 models, which the PR was written alongside, makes exactly that argument.

Two things are different from the hosted version. First, llama.cpp does not search anything. On Anthropic's API you can include a built-in regex or BM25 search tool and the API runs it; in llama.cpp the search tool is yours, and the PR's tests define an ordinary tool_search function the client implements. That matches Anthropic's "custom tool search" mode rather than its default one.

Second, llama.cpp itself does almost nothing with the flag. The C++ changes parse defer_loading and tool_reference, carry them through, and hand them to the model's chat template. Whether a tool is actually withheld, and what a reference turns into, is decided by the Jinja template that ships with each model. The PR lists three models that support it: LFM3-1.7B, LFM3-VL-2B and GLM-5.1.

Same request, three different prompts

To see what that means in practice, we rendered one conversation (a search tool, one deferred get_weather tool, a search call and its result) through three current chat templates from Hugging Face, using Python's Jinja2 rather than llama.cpp's own template engine, so treat the details as an approximation of what the server does.

• LFM3-1.7B behaves as documented. The deferred tool is absent from the system prompt, and the reference is expanded into the full definition inside an <additional_tools> block in the tool response, whether or not the result also contains text.
• GLM-5.1 withholds the deferred tool too, but only expands references if the first content part of the tool result is a tool_reference. A result that begins with a short text part, such as "Found 1 tool." followed by the reference, which is the pattern Liquid's documentation shows, renders as just the text. The model sees that a tool was found and never sees its schema.
• Qwen3-8B, whose template predates the feature, ignores the flag: the "deferred" tool sits in the system prompt like any other, defer_loading: true included, and the search result renders as an empty <tool_response>.

None of these cases raises an error. LFM3's template is the only one of the three that fails loudly, and only for a reference to a tool missing from the request.

So the feature is real and the prefix-caching argument for it is sound, but it is a contract between your client and one model's template, not something the server guarantees. If you build tool search on top of llama.cpp, check the rendered prompt for the model you actually run (the server's /apply-template endpoint shows it), put references first in tool results if you want them to work on GLM-5.1, and don't assume a model whose template never heard of defer_loading will save you any context.

We covered LFM3's architecture and benchmarks here.