Claude Fable 5.2 and Opus 5.5 continue little-studied bacterial proteins; GPT and Gemini don't, Xinming Tu finds
Xinming Tu, a PhD student at the University of Washington who also works at Phylo, published a test on November 17 that asks language models to continue protein sequences almost nobody has written about. The post, "Claude was trained on protein sequences at scale", reports that Claude Fable 5.2 and Claude Opus 5.5 can do it, and that the GPT and Gemini models and seven open-weight models Tu ran cannot. Code, test sets and every model reply are in the GitHub repository XinmingTu/LLM-bioseq-probe.
Claude was trained on protein sequences, at scale.
We gave 20 language models the start of little-studied bacterial proteins and asked for the next amino acids. Only the latest Claude models predict them. Not GPT, not Gemini, not seven open-weight models. pic.twitter.com/lQ9CKdu4OA
The test
The question came from earlier results in which frontier models looked good at protein tasks: Atharv Jayprakash showed GPT-6 Astra ranking SOD1 variants about as well as ESM-2, for example. Did they learn from sequence databases such as UniProt, or from papers about those proteins? Famous proteins can't separate the two, since a sequence like GFP's is printed everywhere. So Tu picked proteins with nothing written about them.
The main set is 668 bacterial and archaeal proteins from 167 Pfam "domain of unknown function" families. Each family has at least 300 members, no solved structure and at most three hits in Europe PMC, and each protein has no other UniProt sequence within 90% identity. Every protein is cut at five random points inside its domain. The model gets the residues before the cut, writes the next ten amino acids, and only the first one is scored. The same question is then asked on a shuffled copy of the protein: same letters, random order. The difference between the two accuracies is what the model knows about the sequence beyond how common each amino acid is.
The tests and their pass thresholds were written down in docs/preregistration.md on November 16, before the confirmation run. Opus 5.5 and Opus 5 were the primary models; Fable 5.2 and the GPT-6.2 models were added later.
Results
| Model | Real proteins | Shuffled | Gap, points |
|---|---|---|---|
| Claude Fable 5.2 | 54.1% | 9.0% | +45.1 |
| Claude Opus 5.5 | 34.9% | 7.5% | +27.4 |
| Claude Fable 5.1 | 21.4% | 6.6% | +14.9 |
| Claude Opus 5 | 17.5% | 7.7% | +9.7 |
| GPT-6.2 Sol | 10.7% | 8.3% | +2.4 |
| GPT-6.2 Astra | 10.0% | 8.8% | +1.1 |
| Gemini 4 Carbon | 8.3% | 7.5% | +0.9 |
Real and shuffled are accuracies over all cuts; the gap is the mean of the per-family gaps. It starts to show in Opus 4.7 (+8.9); Opus 4.5 and 4.6 are near zero. The preregistered threshold asks for the lower end of the 95% confidence interval to sit above 10 points. Opus 5.5 and all three Fable models (5, 5.1 and 5.2) clear it; Opus 4.7, 4.8 and 5 have smaller gaps that fall short. The open-weight models (DeepSeek V4.1 Flash, Qwen 3.8, Kimi K3.1 Pro, GLM-5.3, MiniMax M3, gpt-oss-120b and Nemotron 3 Ultra) were also given the raw FASTA text to continue with no instruction, the format closest to how they were trained, and stayed between 0 and +1.9.
Fable 5.2 was run through OpenRouter; the repository notes that Anthropic's API serves it only to workspaces that retain data.
I recomputed the headline numbers from the published replies in results/outputs/confirm. They match: +45.09 for Fable 5.2, with a positive gap in 164 of 167 families, and +27.36 for Opus 5.5, positive in 158. I also checked what the low numbers amount to. Guessing the most common amino acid in the prefix gets the next residue right 9.9% of the time on the same 3,340 real cuts. GPT-6.2 Sol's 10.7% is about that, so on these proteins the GPT and Gemini models do roughly what a frequency count does.
Why Tu reads it as training data
The gap grows with family size, as it does for protein language models trained only on sequences. On 257 families, including 90 smaller ones down to 20 proteins, Opus 5.5 goes from +8 to +37 points and ProtGPT3-10B from +3 to +37. Fable 5.2 starts at +20 and levels off near +45 from about 300 proteins per family. That ceiling is why it fails the preregistered family-size test on the main set (Spearman 0.00), where every family has 300 members or more.
Papers don't explain it: in the 128 families with no papers at all, the gap is the same as on the whole set. Reasoning doesn't either. The API reports thinking tokens on 9% of Fable 5.2's answers and 36% of Opus 5.5's, and the answers without thinking are as accurate.
Given the same prefix as the protein language models, Fable 5.2's +45.1 is ahead of ProGen3-3B (+32.5) and ProtGPT3-10B (+27.7), and Opus 5.5 is level with ProtGPT3-10B. ESM models, which are trained to see both sides of a residue, score lower on left-to-right continuation, so that part of the comparison favours the autoregressive models.
DNA behaves differently. Opus 5.5 continues human and mouse coding sequence (25 exact 20-nucleotide continuations in 500 human windows) but not introns, promoters, intergenic DNA, or the bacterial genes that encode the proteins it knows.
Memorized or learned?
The November 18 update adds 56 proteins, collected by Leo Tianlai Chen, that were first made public in August 2026, after an assumed July 1 training cutoff, and have no earlier relative at 50% identity or more. Both Claude models still predict them, at about half their main-set gap on the same 31 families: Fable 5.2 +22.1 against +40.2, Opus 5.5 +13.4 against +27.1, GPT-6.2 Sol −0.4. With 56 proteins only differences of about 14 points are reliably detected, and the cutoff date is Tu's assumption.
Jan Kosinski tested the memorization reading the same day, with Tu's published replies and his own Fable 5.2 runs through headless Claude Code, which he says score about 10 points below Tu's API runs. In a thread whose analysis posts he labels as written by Opus 5.5, he reports that Fable's gap falls from 54 points to 36 as close UniProt relatives disappear, and to 22 on the new proteins, with ProGen3 following the same pattern. On 48 new proteins that are 70–95% identical to an older one, where the new protein changed a residue, Fable gives the old residue 59% of the time and the new one 7%. His reading is that Fable 5.2 "behaves like one trained on UniProt", with no more memorization than protein language models show.
@TuXinming and Leo Chen report that Claude Fable 5.2 predicts the next amino acids of obscure bacterial proteins. I suspected it recalls database sequences and asked Opus 5.5 to check. All predictions are by Fable 5.2. Opus 5.5 wrote the [AI-written] posts, not much revised.
What the test doesn't show
Tu's docs/findings.md lists what it can't tell: whether the sequences came from a deliberate corpus or from crawled database pages, in which training phase they entered, or that other labs' models never saw sequences. Anthropic hasn't said anything about protein data in Claude's training that I could find, and neither its official accounts on X nor the staff who usually announce Claude changes had posted about proteins between November 16 and 11:00 UTC on November 18.
The test also measures next-residue prediction, not structure. Sergey Ovchinnikov's prompt that asks Fable 5.2 to predict residue contacts and build a 3D model works on ubiquitin. Kosinski tried it on three bacterial proteins with no relative in the PDB and got long-range contact precision of 0.04 to 0.07 on two of them and 0.07 to 0.33 on the third, against 0.18 to 0.49 on well-studied proteins of the same size and fold class. That is three proteins against three, scored against AlphaFold models rather than solved structures, and a contact-map pilot of Tu's own was added to the repository on November 18. So far the next-residue result is what has held up under rechecking: on families nobody has studied, Fable 5.2 predicts the next amino acid better than any open protein language model Tu ran, and Opus 5.5 does as well as ProtGPT3-10B. On folds with no solved relative, the only test so far, on three proteins, found little.
Comments
No comments yet. Be the first.