vLLM v0.34.0's upgrade notes get the KV-cache rollback advice wrong
The vLLM v0.34.0 upgrade notes offer two ways to keep the old KV-cache sizing. One of them, --gpu-memory-utilization 0.9, leaves the new allocator running with a lower memory budget. To switch back to the old allocator, use --no-enable-extensible-kv-cache. Upgrade notes
The release, published on November 16, changes when vLLM decides how much memory the cache can have. On supported CUDA and ROCm configurations, it now waits until CUDA graph capture and warmup have finished. Keeping a utilization setting from an older deployment can leave you with less cache than before.
I checked both flags against the configuration and allocation code in the v0.34.0 tag. The GPU measurements below come from the maintainers and issue reporters; I have not reproduced their benchmarks.
Two flags with different jobs
The distinction is visible in CacheConfig. An explicit utilization value is returned as given. If you leave it unset, the default depends on whether the extensible cache is enabled:
if self.gpu_memory_utilization is not None:
return self.gpu_memory_utilization
if self.enable_extensible_kv_cache:
return 1.0
return self.DEFAULT_GPU_MEMORY_UTILIZATION
The old default constant is still 0.92. Setting utilization to 0.9 changes the budget available to the current vLLM instance. The allocator selection remains controlled by enable_extensible_kv_cache. Tagged configuration code
There is a measured consequence in Nick Hill's original PR. Across eight models on GB200 GPUs with 189 GiB each, the new defaults added 4.9–11.3 GiB of cache per GPU, enough for 4–10% more tokens. Qwen3-8B went from 150.2 to 161.6 GiB. But when the same new allocator was capped at an explicit 0.92, every model got less cache than with the old sizing, by 0.6–3.7%. The new calculation accounts for allocations that the previous one left to the spare memory outside its budget. Capacity measurements
So retaining a familiar percentage does not preserve the old calculation. A cap of 0.9 lowers the budget further. It may be appropriate for sharing a GPU, but it is a poor instruction for someone trying to roll back the allocator.
vLLM even warns about this at startup. When an explicit value between 0.7 and 1.0 withholds at least 1 GiB from the cache, the warning recommends leaving the setting unset unless something else needs that memory. Warning condition
The cache grows without moving its addresses
The old order had a constraint. CUDA graphs capture memory addresses, so the cache had to exist before capture. vLLM estimated how much space the graphs and later temporary buffers would need, then sized the cache around that estimate. The default utilization of 0.92 left an 8% allowance for the estimate being wrong, about 15 GiB on a 192 GiB GPU.
The new allocator reserves the cache's virtual address range first. Physical memory is attached only to the blocks used during capture and warmup. Once those steps have run, vLLM measures the remaining memory and maps more physical pages into the reserved range. The addresses do not change, so the captured graphs stay valid. Allocator design
The new default of 1.0 still leaves room for serving. The tagged code keeps free the larger of the measured temporary-allocation peak and the measured growth in PyTorch's memory reservation. It also applies a safety margin, at least 1 GiB on CUDA or 3 GiB on ROCm, increasing with the temporary peak. That reserve is part of the calculation even when you set no percentage cap. Sizing calculation
The PR reports engine initialization 21–34% faster, excluding weight loading, because the separate graph-capture pass used for estimation goes away. Those figures describe startup. The release notes also claim a MiniMax-M3 cache increase from 433,000 to 672,000 tokens on H200 and 6–18% higher per-token decode latency for some H200 models at concurrency 128. I found no linked benchmark for those two H200 figures. PR measurements · Release claims
Warmup has to measure the right work
Two bugs found before the release show what can go wrong with this approach.
On a 32 GB RTX 5090, a user running nvidia/Qwen3.8-27B-NVFP4 at utilization 0.85 got just 1.46 GiB of cache. One 36,000-token request needed 3.35 GiB. The allocator had counted FlashInfer's one-off autotuning peak as memory that serving would keep needing. Excluding that peak restored enough cache for startup in the reported tests. The fix shipped in v0.34.0. Report #66193 · Fix #66197
On eight H200s, Qwen/Qwen3.5-397B-A17B-FP8 failed on its first 8,192-token step in the release candidates. PyTorch had 1.50 GiB reserved but unallocated at the failure. The measured allocation peak had missed the extra reservation needed around large workspaces. A proposed fix raised CUDA's minimum margin to 3 GiB. The merged fix instead measures reservation growth and keeps room for it. Initial report and proposal · Merged fix #67296
An open report on November 17 showed the measurement still leaving an unexplained gap. With FlashInfer's PCIe IPC all-reduce enabled, a nightly build allowed 1.758 GiB per GPU for a 1.28 GiB workspace. On the same reported setup, the old allocator at utilization 0.98 ran out of memory during graph capture because the workspace appeared after sizing. Switching allocators can change which failure you encounter. Workspace report #67521
For distributed deployments, every worker also has to agree on the allocator. Another open report described a hang after weight loading when nodes running different builds resolved enable_extensible_kv_cache differently. One branch entered a collective operation that the other workers never reached. Mixed-build report #67763
When a memory cap still belongs in the configuration
If another server or training job needs the same GPU, an explicit utilization value remains useful. Warmup cannot budget for allocations another process will make later. The configuration documentation also names unified-memory devices, weight-transfer buffers and KV connectors with their own device buffers as reasons to leave space. Configuration documentation
Weight reloads are one such case. Updating layers can temporarily require several times the checkpoint size of the largest layer. The default RL update flow pauses serving and clears the KV cache, allowing that memory to be released for the update. Calling reload_weights directly while serving needs headroom from the utilization cap. Without enough space, a reload can warn, fail before loading, or run out of memory. Weight-update behavior
Comments
No comments yet. Be the first.