SpecPilot: a Claude Code agent pushed speculative decoding to 10.87× over autoregressive. We checked its lossless 12-bit weight format on three models

SpecPilot (arXiv 2611.02501, posted Nov 2) is a paper by Kunjun Li, Wenhao Chai and Yihao Liang (equal contribution), with Yuezhou Hu, Shuyao Xu, Yi Pan, William X. Zheng, Kannan Ramchandran, Kurt Keutzer and Jiantao Jiao, from UC Berkeley, Princeton, Stanford and NVIDIA. It lets a coding agent (Claude Opus 5.5 running in Claude Code) tune a whole speculative-decoding serving stack for one model on one GPU, and reports that the result runs Qwen3.8-27B with a DFlash2 drafter at 10.87× the speed of plain autoregressive decoding on an RTX PRO 6000, up from 4.46× for the stock released configuration. On H100 the numbers are 8.86× and 9.41×, or roughly 416–435 tokens per second against 49.9 for autoregressive decoding.

The code is out under Apache-2.0. We read the paper and the repository, and re-implemented the most reusable piece, a lossless 12-bit weight format the agent arrived at, on three open models. That part holds up. The headline multiple needs more context than the abstract gives it.

What the agent actually does

The loop is not "ask an LLM for a faster kernel". The authors give the agent a cost model of speculative decoding (speedup as accepted tokens per step divided by the relative cost of a step, split into a ceiling, kernel and overhead terms), run several agents in parallel with one direction per GPU, and keep a structured memory of about 140 entries. A controller the agents cannot edit decides what gets promoted: a change is kept only if the 95% lower bound of its gain beats 1% plus twice the measured noise of the stock system. Correctness is checked by token forcing, which rejects a system whose mismatch rate against the reference exceeds stock's by more than 0.5 points. Finally, each promoted system is re-measured on a sealed set of requests the agents never saw.

Out of that come several mechanisms that are useful on their own:

• W12, a lossless 12-bit storage format for BF16 weights, decoded in registers inside a stream-K GEMM. The paper reports 1.19–1.28× faster autoregressive decode and 14–22% less weight memory, and calls it the first lossless format whose GEMM beats cuBLAS (by 1.17–1.47×). Earlier lossless schemes such as DFloat11 are, by the paper's own comparison, 4.8–6.7× slower than cuBLAS.
• Two-level verification: a cheap pass over an 8-bit view nested inside W12, followed by one exact pass. The cheap pass agrees with the exact top-1 token 98.7% of the time.
• A certified INT8 LM head, best-first tree verification kernels for hybrid Gated DeltaNet models, and three batching mechanisms (a shared node budget, per-request drafter block size, and prompt admission).

The big jump is not one breakthrough. The paper's appendix (Table A3) shows it as about 15 small promoted changes stacked on top of each other, with two-level verification as the largest single step. Each deployment cost 20–86 GPU-hours and 7–19 wall-clock hours of agent time.

Our check: W12 is real and cheap

W12 is simple enough to verify without a GPU. BF16 has 8 exponent bits, but within one output row of a weight matrix almost all weights fall in a narrow range of magnitudes. W12 picks, per row, the window of 16 consecutive exponents that covers the most weights, and stores each weight as sign, a 4-bit offset into that window and the 7 mantissa bits: 12 bits. Anything outside the window, plus zeros and denormals, goes to a per-row escape list stored exactly.

We wrote a numpy encoder from the description and the repository's kernel code and ran it over every 2D weight matrix (excluding embeddings and LM head):

The paper reports 12.013–12.020 bits per weight and escape rates of 6.9e-5 to 1.21e-4 on its own models, so our numbers land in the same place (our bits figure uses our own guess at per-row metadata, so treat the second decimal loosely). An explicit encode–decode round trip on a 896×4864 matrix from Qwen2.5-0.5B came back bit-identical. For comparison, entropy-coding the exponents in the DFloat11 style would get these models to about 10.6–10.9 bits per weight, so W12 deliberately gives up roughly 1.2–1.4 bits of compression to keep decoding to a few register operations. That trade is the whole point, and it is the piece of SpecPilot most likely to be picked up by other serving stacks, independent of the agent story.

What to keep in mind about the 10.87×

• The baseline is the stock configuration, not the strongest system. Against DDTree, a recent tree-based speculative decoder that reaches 4.82× over autoregressive in the same setting, SpecPilot's gain is 1.84×. Still large, but the 10.87× figure folds in both the drafter and everything the agent did.
• The benchmark is small. In single-request settings the repository benchmarks the first six items of each dataset: 30 requests with 2,048 new tokens, greedy decoding with thinking on. The sealed re-measurement on unseen requests is encouraging (gains over stock of 2.13–2.44× for single requests and 1.85–3.11× batched, close to the development numbers), but it is still the authors' own harness.
• "Lossless" means up to floating-point reduction order. Outputs are not bitwise identical to autoregressive decoding, and the acceptance rule tolerates 0.5 points more token mismatch than stock. In the README's batched sanity checks, autoregressive decoding scores 177–178 of 240 while both stock speculative decoding (170–173) and SpecPilot (167–172) are a few points lower. The authors call this a sanity check, not an accuracy evaluation, and with 240 items the gap is within noise, but it is worth watching.
• The discovery loop itself is not released. The repository ships the six final engines (a pruned SGLang fork plus per-setting overlays and benchmark scripts), and the README reports the authors reproducing 1.94–2.52× over stock from a fresh clone on H100. The agent loop, the controller, the memory and the token-forcing scorer are not in it, so the claim that an agent found these optimizations cannot yet be reproduced, only its results.
• There is no limitations section in the paper.

Why it matters

Speculative decoding is one of the few inference tricks that speeds up single-user latency without changing the model, and serving stacks have been shipping it with conservative defaults; we covered one of the resulting correctness fixes in SGLang's cached-continuation bug after constrained speculative decoding. SpecPilot's argument is that the defaults leave a lot on the table, and that a tightly-controlled agent can recover it per model and per GPU for tens of GPU-hours. The final engines are runnable today. The more general claim, that this loop transfers to new models without a human in it, waits on the release of the loop itself and on someone outside the author group running the engines.