Stanford re-tests 50 famous LLM findings on 20 models: 44% have expired, and the ones that last are about sycophancy and self-preference

Owen Queen and James Zou of Stanford have re-run 50 widely cited findings about how large language models behave, on 20 models released between July 2024 and September 2026, and most of the findings did not survive. In "The shelf life of LLM findings", published on November 2, they classify 22 claims (44%) as expired, 17 (34%) as weakened, 8 (16%) as persisting and 3 (6%) as stronger than when they were first reported.

The test is simple to describe. For each claim the authors took the key metric from the original paper, checked that they could reproduce the paper's result on the original model or a close substitute, then ran the same setup on a panel of models from eight developers and fitted a trend line against release date. Everything is put on one scale, where 1 means the effect is as strong as in the paper and 0 means it is gone. A claim counts as expired once the fitted effect falls to a fifth of the original or less and stays there through September 2026.

What faded

The clearest example is the one people still repeat most. In 2023 Walters and Wilder asked ChatGPT, then running GPT-3.5, for short literature reviews and found that 55% of the references did not exist. Queen and Zou reproduced that (GPT-3.5 Turbo invented 59% of 210 references across 42 topics), then sent the same prompts to current models: Claude Sonnet 5.5 fabricated 1.37% of its references and GPT-6.1 Sol none.

The other expired claims are mostly capability failures with a right answer: counting the letters in "strawberry", arithmetic in unfamiliar bases, losing facts in the middle of a long context, dropping a correct answer when the user pushes back, overthinking simple maths. The authors' guess is that these are exactly the things labs can measure and train against, and that some high-profile failures were probably patched directly in post-training.

Not every expiry is a capability story. Harsher judgments of speakers of African American English, stereotyped personas of Black women and more non-answers in Arabic than in English are all classified as expired; for the dialect effect, the fitted trend actually reverses direction.

What stayed

The findings that held up are mostly about preferences rather than errors. Models still agree with and flatter the user, still answer when they should abstain, still lean left on political tests, still give near-identical answers to open-ended prompts. The three claims that got stronger are LLM judges favouring their own writing, praise that follows the user's stated preference, and non-uniform "random" numbers, with fitted effects about 2.4, 1.9 and 1.7 times the original. The self-preference result starts from a small base, a 4.6-point gap in the original GPT-4 study, but GPT-6.1 Sol's effect is 2.88 times that.

Some structural limits weakened without disappearing. The reversal curse ("A is B" learned, "B is A" not) is at about half its original strength; Claude Sonnet 5.5 still gets roughly one two-hop question in ten wrong even when it answers both sub-questions correctly, against 43% for ChatGPT in the original paper. Basic prompt injection is at about a quarter.

The fine print

The panel-wide verdict can hide a lot. Citation fabrication is "expired", yet Qwen3.7-Flash invented 69% of its references in the same test. Among models released in July to September 2026, the number of expired claims ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol, so "newer" is not the same as "fixed".

The authors are also candid about the weak spots. Only 50 claims, so the aggregate estimates have wide intervals. Four 2024 models anchor the early part of every curve. Retired models had to be replaced by substitutes in 23 of the 48 reproduction checks. Every model ran at its provider's default settings, including default reasoning effort, so the "overthinking" result says nothing about models with thinking turned up. And the blackmail finding from Anthropic's agentic-misalignment report is listed as expired around September 2026, but the authors call the support for that verdict weak and note that several recent models still blackmail frequently.

One more detail is worth noticing: the reproduction pipelines were built with Claude Code and Codex, which is how a two-author project got through 50 papers. The study is an interactive web write-up with a BibTeX entry rather than an arXiv paper, and we found no link to its code or raw outputs on the page.

Why it matters

The practical conclusion is about how we cite things. A finding about LLMs describes particular models at a particular date, and the authors ask papers to report model, version and date with every result and to ship prompts and analysis code so the test can be rerun. They point at a concrete case: ICLR 2026 desk-rejected papers with hallucinated references and ICML 2026 lists them as grounds for rejection, policies built on a failure rate that current frontier models mostly no longer have.

Ethan Mollick's summary on X was blunt: recent models no longer invent citations and both hallucination and stereotyping have dropped, "but they are still sycophantic & lean left."

The useful reading is narrower than "old LLM research is wrong". In this sample, things models used to get wrong were trained away within about a year and a half (the 22 expired claims lasted 15.7 months on average, median 14.7). Things models are inclined to do, especially as judges and as conversational partners, mostly were not. If you use an LLM to grade other models' outputs, that second list is the one to read.