Cloudflare's GLM-5.3 data leak came from an SGLang bug first reported in July as "garbage output"

For almost two weeks this summer, Cloudflare's Workers AI deployment of GLM-5.3 could answer one customer's very long prompt with text from other customers' requests. Cloudflare's coordinated disclosure, published October 26, counts 155 affected responses between August 27 and September 9, all to prompts longer than 681,280 tokens. The root cause was a bug in SGLang, the open-source inference engine Cloudflare uses, and that bug had been described in detail on SGLang's public tracker in July, as a quality problem rather than a security one.

Two mistakes that met

Cloudflare served GLM-5.3 with a 1,048,576-token context window, but the KV cache pool on its prefill GPUs had room for only 681,280 tokens. The company says it knew about the mismatch before launch. It expected the engine to return an error for anything that did not fit, and its long-context tests came back with HTTP 200, so it did not treat the gap as a problem. In its own words, it "only validated HTTP response codes and engine performance at long contexts, without validating model output and coherence".

The engine did not return an error. Workers AI runs GLM-5.3 in disaggregated mode: prefill GPUs process the prompt and ship its KV cache over RDMA to separate decode GPUs, which generate the answer. When a prompt was too long for the pool, SGLang's scheduler marked it as rejected and swapped the prompt for a one-token placeholder, a trick that works on a single node, where the placeholder drains through the batch and the client gets an error. In disaggregated mode, however, both admission queues checked whether a request was finished, not whether it had been rejected. The rejected request went through anyway. Prefill computed KV for one token, sent one page, and reported the transfer complete. Decode had already reserved pages for the full prompt length and attended over all of them.

KV cache pages are not zeroed between requests, which Cloudflare describes as common practice and necessary for performance. The pages this request never wrote still held earlier users' prompts and partial generations. The client received a coherent reply with prompt_tokens matching its long prompt. The one outward clue was speed: the answer arrived faster than a prefill of that length should take.

The bug report that called it garbage

SGLang issue #30233, opened July 6, walks through the same mechanism step by step: an over-length input in PD disaggregation mode, set_finish_with_abort truncating the prompt to one token, the aborted request still entering the prefill queue, a one-token KV transfer, and decode starting on a reservation for 284,937 tokens. It even includes a proposed patch. Its title says the result is "garbage output", and it describes the untouched slots as "uninitialized KV cache".

That description was reasonable on a test machine running one user's requests. On a shared multi-tenant deployment, "uninitialized" meant "left over from someone else". The issue drew no reply from maintainers and was closed automatically by a bot for inactivity on September 5, nine days into Cloudflare's exposure and three days before Knostic researcher Gadi Evron reported the leak to Cloudflare through HackerOne. Cloudflare says it found the issue while investigating.

A fix had in fact just landed. PR #36988, labelled as a vision-model change and titled "Retire aborted disaggregated prefill results", was merged on September 6 for a related failure path. It shipped in v0.5.20 on September 18. PR #38935, merged September 16 and released in v0.5.21, rejects such requests at admission, before any allocation or transfer; it follows the approach of Cloudflare's own patch, which was closed in its favour.

What changed afterwards

Cloudflare took the model private within 40 minutes of declaring an incident, rolled its prefill nodes to a configuration whose pool holds about 2.04 million tokens, roughly twice the context length, and restored the model the same day with a patched engine. It says no other Workers AI model was affected, because every other disaggregated deployment had pools larger than its context length, and that it found no evidence of deliberate exploitation.

The upstream work goes beyond this one path. Cloudflare places the incident in a class of KV-cache memory-safety bugs, in which a page is freed, reused or read while something else still has a claim on it, including races where RDMA writes for an aborted request land in pages already handed to another. SGLang has since merged an optional per-transfer checksum (--disaggregation-enable-kv-checksum), turned deferred KV release on by default, and fixed several abort-confirmation paths, most of them in v0.5.21 and v0.5.22. A draft proposal to optionally zero KV pages on release was opened on October 25 and is still open. Cloudflare notes that the checksum and SGLang's KV canary do not yet cover every architecture, GLM-5.3 included.

The most interesting idea in the post is one Cloudflare tried and shelved: tagging every physical KV page with a generation number that changes on reallocation, and checking before each decode read that a request only attends over pages it owns. In its tests this cost under 1% in throughput and tail latency, but the prototype was tied to specific attention backends and was disabled because of reliability issues. Checksums only check the handoff. Page tags would catch a stale read whenever it happens, which is the property a shared inference server actually needs.

For anyone running SGLang with disaggregated prefill and decode, Cloudflare's advice is concrete: compare the KV pool size the engine logs at startup with the configured context length on every node after any change to weights, quantization, parallelism or memory fraction, and upgrade to v0.5.22 or later. Sysnote's earlier note on another v0.5.22 cache-correctness fix suggests that release is worth taking for more than one reason.