Fireworks AI breach: attackers got the data its "zero data retention" never covered, and Notion AI chats were in it

Fireworks AI's November 9 update on its October breach ends with a reassuring line: "We have not detected any impact to Inference, Training SDK and Nexus, which are all zero data retention (ZDR) by default." That is probably true, and for many of its customers it is beside the point. What the attackers reached was everything Fireworks does keep: secrets attached to evaluation jobs, datasets, partially trained model weights, and account and billing records. Fireworks' own documentation lists exactly those features as the ones that store customer content.

Notion is the only customer we found that has published what that meant in practice. In an advisory updated November 9, it says Notion AI chat transcripts used for its Insights feature were in the stolen data, affecting fewer than 1% of Notion workspaces.

Two intrusions, one bulletin

The first Fireworks bulletin, published October 9, described a narrow incident. On October 6 Fireworks found "unauthorized use of internal Fireworks credentials," which an outside party used to read environment variables in evaluation jobs in "a limited number of customer accounts" — in other words, API keys and tokens customers had stored as secrets. At that point Fireworks said it had "not detected evidence of unauthorized access to customer models, datasets, training data, personal information, or inference traffic," and added that "Fireworks does not store prompts or outputs by default, as our platform is ZDR."

The November 9 update adds a second way in. A third-party cloud provider that hosted Fireworks R&D machines had its own security incident on October 23. The next day the intruder used it to reach Fireworks development machines and "run an unauthorized workload that led to additional customer data impact." Fireworks says it detected that the same day and has seen no further successful access to its customer platform since October 25. It does not name the cloud provider.

The "affected internal application," the update says, gave access to resource metadata (identifiers, names, descriptions), billing information (addresses, invoice and payment history, balances and credit usage; no card numbers, which Fireworks doesn't store), and, "for certain customers, a subset of incomplete model weights and datasets." Fireworks ties the attack to a financially motivated group that sells stolen LLM access through illicit routing gateways, the "LLMJacking" economy that Google Threat Intelligence Group and Mandiant described on November 2, with fraudulent bills above $100,000 a day in some incidents.

So the October bulletin wasn't contradicted so much as overtaken: the dataset access came through a second intrusion two weeks after it was published. But the update arrived on November 9, more than two weeks after Fireworks says it detected that activity.

What "ZDR by default" covers

Fireworks' data-handling page defines its default ZDR narrowly and precisely: prompts and generations for open models exist "only in volatile memory for the duration of the request" and are not written to persistent storage. That is a promise about the request path.

Its separate Zero Data Retention policy page is more candid about the rest of the platform. "Most of Fireworks never stores prompts or generations. A few features do store customer content, such as stored responses, batch jobs, and training." The policy that blocks those features is an Enterprise option, "Off by default," and when enforced it rejects, among other things:

Put that table next to the breach. The October intrusion took secrets from evaluation jobs. The October 24 intrusion took datasets and incomplete model weights. Every category of customer content Fireworks has said was accessed sits in the "Rejected" column — the features that, by Fireworks' own description, keep data at rest. Turning on the policy also "doesn't delete anything already stored on the account," so it would only have helped customers who had never used those features.

Fireworks' response points the same way. It has "disabled self-serve dataset uploads and downloads used by Batch Inference and dataset-based training," while noting that the Training SDK "remains available with ZDR, i.e. no training data stored at rest." Batch inference, for reference, takes a JSONL dataset of up to 80 GiB and writes results to an output dataset; the Batch API docs pitch it for data labeling, distillation and large-scale evaluation. That is the kind of workload that leaves customer text sitting in storage.

Notion's part of it

Notion's advisory and its longer help-center notice say it used Fireworks two ways: as a general inference provider for open-weight models inside the product, and for "classification jobs to help customers understand their own AI usage through a feature in Notion called Insights." The leaked transcripts came from the second use. Notion doesn't say which Fireworks product ran those jobs. But inference traffic, by Fireworks' account, was not touched, and a classification job over a pile of transcripts is a stored-dataset workload by nature.

The timeline also explains why Notion's quick reaction didn't help. Fireworks first told Notion on the evening of November 2 that "a small set of data" had been accessed; Notion found it was synthetic test data. By the morning of November 3 Notion had disconnected from Fireworks, "deleted all data in the Fireworks system" and, according to the help-center FAQ, "halted all training runs." On November 6 Fireworks told Notion customer data might also have been taken, and confirmed the dataset on November 8. If the access happened during the October 24 intrusion, as the Fireworks update suggests, the data was copied more than a week before Notion deleted it.

Two other details are worth noticing. The help-center FAQ mentions that Notion previously used data volunteered by customers in its Learning and Early Access Program (LEAP) "to experiment with training task-specific models on Fireworks" — a reminder that a vendor relationship that starts as inference can accumulate stored data over time. And the two Notion pages describe the transcripts slightly differently. The help-center notice says the exposed chats were "unattributable," meaning not labeled with a user or workspace. The advisory says they were unattributable "for the vast majority of impacted workspaces," which leaves room for some that were not. Notion hasn't explained the difference. Either way, the content of a chat can identify its author even without a label, which is why Notion tells affected customers to rotate any credentials pasted into Notion AI.

How affected users find out has also drawn complaints. Enterprise workspace owners get a CSV of the exposed data plus an annotated version that maps transcripts to users. Owners on other plans get a page with a link they must share with affected members, who were not emailed directly. In the Hacker News thread one workspace owner quoted the notice that conversations from two members "may have been affected" and that "they have not been proactively emailed."

Notion is not the only customer involved. A Fireworks spokesperson told Bloomberg Law the company is working with customers "including Notion, Cursor and others." Neither Cursor nor any other customer had published an impact statement when we checked on November 10.

What to check if you use an inference provider

The lesson is not specific to Fireworks. Most inference providers now advertise zero data retention for their main API, and most also sell batch jobs, fine-tuning, evaluations and hosted datasets that cannot work without storing your data. The ZDR label usually describes the first group.

If you use any of those features with any provider, it is worth knowing:

• which of your workloads leave data at rest (input and output datasets, fine-tuning data, evaluation configs, uploaded or partially trained weights), and whether you delete them when a job finishes;
• whether the provider offers an account-level policy that blocks persistence, and whether it is actually switched on — at Fireworks it is Enterprise-only and off by default;
• which secrets you have handed to the provider's jobs. Fireworks' October bulletin made the point itself: removing a stored secret from its dashboard "does not invalidate the underlying credential." Rotate it with whoever issued it.

Fireworks says its investigation, run with Palo Alto Networks Unit 42, is ongoing and "may reveal additional information." It still hasn't said how many accounts were affected, which cloud provider was breached, or how the internal credentials behind the first intrusion were obtained.