SGLang fixes cached continuations after constrained speculative decoding

SGLang 0.5.22 fixes a correctness bug that could make hybrid linear models resume from the wrong cached recurrent state when speculative decoding and structured output were combined. The October 19 release credits Baseten and GitHub user Wirick with finding it.

The interesting detail is where the two features disagreed. Speculative decoding verifies proposed tokens in batches; an output grammar can stop generation before all the verified tokens are retained. The scheduler’s checkpoint-boundary calculation used the draft-acceptance count even in that grammar-truncated case.

The fix, merged October 10, makes that branch count the tokens actually retained, with an adjustment for non-draft tokens outside the tracked verification window. The released scheduler code uses that count to determine whether the request crossed a boundary at which recurrent state should be tracked.

The accompanying regression test checks more than whether the first answer obeys its format. It runs Qwen3.5-0.8B with NGRAM speculative decoding, an XGrammar regex and the extra_buffer Mamba cache strategy. Prompt lengths sweep across a 256-token checkpoint boundary. A follow-up is then scored once with the cache populated and again after flushing it; the test requires the token log probabilities to remain close and verifies that at least one run actually reused the checkpoint.

That comparison captures the practical failure: a constrained answer can finish successfully while leaving a cached state that changes how the next turn is evaluated. The test checks continuation probabilities, where such a mismatch becomes visible. This is evidence from the project’s implementation and regression test; Sysnote has not independently run the GPU reproduction.