Google's Sleep framework lets LLMs consolidate memories offline, but its GPT-4 win compares fine-tuned and zero-shot models

Google Research has published a blog post on "Sleep", a framework that lets a large language model alternate between an active phase, where it takes in new information, and an offline phase, where it writes that information into its own weights and trains itself on synthetic practice data. The October 27 post pitches it as a way to get continual learning without catastrophic forgetting, and claims wins over in-context learning, fine-tuning, GRPO and, on one long-context benchmark, GPT-4.

The research itself is not new. The paper, "Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories" by Ali Behrouz, Farnoosh Hashemi, Adel Javanmard and Vahab Mirrokni (Google and Cornell), went up on arXiv on June 2 and was revised on July 10; a note in the paper says a version has been on OpenReview since September 2025. What the blog adds is a fresh audience. What the paper adds, for anyone who reads past the summary, is a clearer sense of how large the gains are and what they were measured against.

How "sleep" works

The framework has two stages, both of which happen while the model is not serving requests.

Memory consolidation. New information first lives in fast-changing, short-term components, including the context window. During sleep, the model unlocks extra low-rank experts in a Mixture-of-Experts layer and distills the short-term material into them, leaving older parameters alone. The authors call this "knowledge seeding": on-policy distillation (the model learns from a teacher's token distributions on sequences it samples itself) combined with an RL-based imitation objective that rewards completions for semantic similarity and for Levenshtein distance to the teacher's text. Afterwards the short-term parameters are reset.

Dreaming. The model generates synthetic training examples with its router deliberately sending tokens to random, non-standard experts, filters those "dreams" with gradient-based data selection, fine-tunes on the best ones and checks whether downstream performance improves.

Strip away the neuroscience vocabulary (NREM, REM, synaptic pruning) and Sleep is a scheduled post-training loop: periodic capacity expansion, self-distillation and self-generated curriculum training. The blog says as much, noting that the framework "operates entirely within a fixed model architecture and human-defined training rules". Much of the evaluation runs on Hope, the multi-level memory architecture from the same group.

The GPT-4 comparison

The blog's most quotable line is that on the BABILong long-context benchmark, Sleep showed "near-perfect scaling up to 10M tokens, outperforming strong baselines like GPT-4, and Titans".

The appendix explains the setup. GPT-4 and GPT-4o-mini were evaluated zero-shot. Hope with Sleep, along with the small long-context models it was compared to (RMT, ARMT and Titans), was fine-tuned on BABILong using the benchmark's official training protocol. The paper itself says the large models "fail beyond 128K–256K tokens", so at 10M tokens the GPT-4 comparison says more about context windows than about memory consolidation.

The more meaningful result is among the fine-tuned small models: according to the paper, Titans, ARMT and Hope perform comparably up to about 1M tokens, after which Titans and ARMT degrade sharply while Hope stays stable. Titans, it is worth knowing, is an earlier architecture from the same first author.

The math results are real, but small

On AIME 24, AIME 25 and HMMT 25, the blog reports 79.2%, 69.0% and 46.1% for a "sleep-enhanced base model". Table 2 of the paper shows the base model is Qwen3-8B (scores are average@16), and the gains look like this:

Against on-policy self-distillation (OPSD), the strongest baseline, Sleep gains 2.6, 1.6 and 1.0 points. On Qwen3-1.7B the margins over OPSD are 1.6, 0.2 and 1.2 points. The tables report no variance across runs, so it is hard to say how much of a 0.2-point edge survives a different seed. In the ablation table, the variant with the semantic reward removed scores 69.2 on AIME 25, slightly above the full method's 69.0.

The efficiency section makes a two-sided claim. Per training step, SFT is about four times cheaper than Sleep; but to reach Sleep's scores, the authors say, SFT needs 4.3x, 3.6x and 4.8x the wall-clock time on the three benchmarks. The longer SFT runs behind that comparison don't appear in the tables, where SFT on Qwen3-8B actually ends up below the base model on AIME 25.

Where dreaming earns its keep

The clearest evidence for the dreaming stage is on knowledge incorporation, using the SQuAD setup from SEAL (Self-Adapting Language Models). Sleep with a four-level memory reaches 48.9% on single-passage and 46.2% on continued-pretraining evaluations, against SEAL's 46.7% and 43.2%. Remove dreaming and those numbers drop to 35.7% and 36.2%, close to the 33.4% and 32.0% of plain fine-tuning. That ablation says more about what the method is doing than the headline comparisons do: most of the knowledge-incorporation gain comes from the model training on its own filtered synthetic data.

On continual translation, the model learns two languages absent from pre-training, Manchu and Kalamang, one after the other. Here the paper reports that in-context learning largely reverts to pre-trained behavior after the second language, while Hope variants keep much more of what they learned.

What's missing

The paper links no code, and the blog points only to Google Research's general GitHub page. The named base models in the experiments top out at 8B parameters (Llama 3 3B and 8B, Qwen3 1.7B and 8B). And the paper doesn't take up the practical question the blog's framing invites: a model that rewrites its own weights after each "day" of use is a different deployment object from a fixed checkpoint, with consequences for serving, per-user state and evaluation that nobody has worked through in public yet.

Sleep is a reasonable research direction, and its ablations are more informative than its headlines. The "beats GPT-4" line is the least informative part of the package.