Google Research Gives LLMs a Sleep Phase to Retain Knowledge

The problem: static models and forgetting

Google Research says modern LLMs perform well thanks to pre-training and in-context learning (ICL), but once deployed they remain largely static. They operate with a fixed knowledge cutoff date and cannot easily acquire long-term capabilities or adapt to new information.

Updating the model with new data is the obvious fix, but continual parameter updates often trigger catastrophic forgetting, where learning new tasks severely degrades performance on earlier ones. Relying strictly on ICL has its own limit: knowledge stays within the context window, and once the session ends, the short-term information is lost.

The blog post, written by Ali Behrouz and Vahab Mirrokni and published on October 27, 2026, presents the sleep framework as a way around both limits. It is inspired by how the human brain stabilizes memory during rest.

How the wake and sleep phases work

Under the framework, a model works in two states. In the active (wake) phase, it receives and processes inputs and keeps temporary information in fast, short-term memory components. In the sleep phase, it stops processing new inputs and turns its computation toward memory consolidation and self-improvement.

The researchers stress that terms such as sleep and dreaming are borrowed from neuroscience. According to the post, the framework runs entirely within a fixed model architecture and human-defined training rules. During sleep, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its weights using self-generated practice examples.

The sleep process has two stages:

• Memory consolidation: the model expands its capacity using MoE and the Hope architecture, integrating newly unlocked low-rank experts into stable, lower-frequency memory blocks without disrupting older parameters. Short-term memories are then transferred into these long-term parameters through Generalized Knowledge Distillation (GKD) combined with reinforcement learning (RL) based imitation learning. Afterwards, the temporary high-frequency parameters are reset, which the authors call synaptic pruning.
• Dreaming: the model generates synthetic training samples, called dreams. The MoE router randomly activates non-standard experts to mix disparate domains of knowledge. Samples are filtered with gradient-based data selection, and the model then performs supervised fine-tuning on the top-rated ones and checks whether performance on downstream tasks improves.

Theory and reported results

The authors say the approach is backed by theory. For consolidation, they describe knowledge seeding as a structured, low-rank compression problem: the model does not copy every detail of its short-term traces but identifies a compact set of directions that preserve its own on-policy behavior and the semantic qualities the objective rewards. Parameter expansion supplies the capacity to write these memories without overwriting earlier ones.

For dreaming, the analysis frames random expert activation and reward-based selection as ways to improve the geometry of learning. According to the post, this yields a more balanced optimization landscape in which gradients are better conditioned and progress is more stable.

Google Research compared the framework with baselines such as ICL and supervised fine-tuning across class-incremental learning, continual translation, long-context scaling up to 10M tokens, factual knowledge incorporation and mathematical reasoning. In continual translation for unseen languages such as Manchu and Kalamang, the sleep framework kept a near-zero performance drop in multi-stage evaluations, while ICL, Cartridges and SFT suffered severe catastrophic forgetting. On the BABILong benchmark, the method showed near-perfect scaling up to 10M tokens and outperformed baselines including GPT-4 and Titans, according to the researchers.

What it means

The work points to a way for deployed models to keep learning after release without a full retraining cycle and without leaning only on the context window. The reported results come from Google Research's own evaluations, and the source text provided does not cover the remaining results on factual knowledge and mathematical reasoning in detail. Nothing in the post describes a product release or availability.