The model that never stops learning

figure 1 · the diagnosis

A model with anterograde amnesia

There is a neurological condition — anterograde amnesia — where a person keeps every memory from before the onset, and can hold a conversation in the present, but can never form a new long-term memory again. The paper opens with the observation that today's LLMs live in exactly that condition. A deployed model has two memories: the context window, which updates with every token and vanishes when the session ends, and the weights, frozen at the moment the authors call "the end of pre-training." Nothing connects them. What the model reads today never becomes part of what it knows.

click any figure to zoom · scroll to magnify · drag to pan

Brains don't work like this. New experience is consolidated continuously — a fast "online" pass stabilizes fragile traces as they form, and slower offline replay strengthens and reorganizes them for the long term. Crucially, this runs at many speeds at once: brain waves from fast gamma (30–100 Hz, sensory detail) down to slow delta (0.5–4 Hz, consolidation) coordinate which neurons update quickly and which integrate slowly. That multi-speed picture is the paper's engine.

Where the arc left off: DeepSeek-R1 changed the training signal, and Mamba changed the architecture. This entry questions something quieter and more radical — the assumption that a model is finished when training ends.

figure 2 · the core idea

Layers stack in space — learning stacks in time

The paper's central move is a change of coordinates. Stop describing a model as a stack of layers, and describe it as a set of nested optimization problems, each with its own data stream and its own update frequency — how often that component actually learns something. Order the parts by that frequency and a hidden structure appears.

Run a Transformer through this lens and the result is stark. Attention adapts to every new token — in effect, update frequency ∞. The projections and MLP blocks were updated during pre-training and then never again — frequency 0 in deployment. Two extreme clocks, and nothing in between — which is precisely the amnesia of figure 1, restated as arithmetic. The title's point: "deep learning" is the flattened image of this nested system, with all the inner clocks hidden — the illusion of a mere stack of layers.

figure 3 · the unification

Learning is memorizing surprise

For the nested view to work, all the parts have to be the same kind of object — and the paper's most elegant section shows they are. Train a layer with backprop and watch what it stores: for each input, the gradient it receives is a local surprise signal — how wrong the layer's contribution to the prediction was. Gradient descent is then an associative memory that maps each input to how surprising it was, compressing those pairs into the weights.

The same definition covers everything else. Attention is an associative memory mapping tokens to tokens, re-solved every step. Pre-training is an associative memory over the whole corpus, solved once — which makes it just in-context learning with a very large context and a very slow clock. One kind of object, one definition, different frequencies. That's the uniform structure the rest of the paper builds on.

figure 4 · the arithmetic

Your optimizer is a memory too — a small, blurry one

Here is the reframe with the most immediate payoff. Momentum keeps an exponential moving average of gradients: m ← 0.9·m + η·g. Viewed as a memory, it's a memory of the loss landscape — and a worked line of arithmetic shows how tiny it is. With decay 0.9, the last 6 gradients account for about 50% of its state (1 − 0.9⁶ ≈ 0.47), and the last 43 for 99%. Everything a billion-step training run learned about the landscape beyond the last ~40 steps contributes less than one percent.

Once the optimizer is a memory, you can design it like one. Give momentum an MLP instead of a single matrix (deep momentum). Swap its Hebbian update for a delta rule so it learns what to forget (delta momentum). Pass its output through a Newton-Schulz map and you have re-derived Muon from first principles. Adam falls out as the optimal memory for one particular objective. The paper's slogan in one line: expressive optimizers and expressive architectures are the same craft.

figure 5 · continuum memory

A spectrum of memory, not two extremes

If the disease is two clocks with an empty middle, the cure is to fill the middle. The Continuum Memory System (CMS) is a chain of MLP blocks, each with its own update period: the fastest folds in what it read every 16 tokens, the next every thousand, the slowest every few million (the paper's example runs chunk sizes 16 → 1M → 16M). Each block accumulates its own surprise while it waits, then consolidates in one step when its clock ticks — short-term memory shading into long-term knowledge with no hard boundary, exactly the gamma-to-delta spectrum of figure 1.

The design has a built-in defense against catastrophic forgetting: knowledge overwritten in a fast block still lives in the slower blocks behind it, and can be handed back — a loop through the time dimension. And it stays cheap: at any given step only the blocks whose clock is due actually update, and within a chunk every token's contribution can be computed in parallel. A Transformer is the special case with one block and a frozen clock.

figure 6 · self-modification

A model that edits its own update rule

The second component works on the fast end of the spectrum. Start from a modern recurrent memory — a Titans-style module that compresses keys and values into its state as it reads. In the standard version, the projections that make keys and values, the learning rate, and the forget gate are all frozen after pre-training. The paper makes each of them a small memory of its own, adapting in-context — and then adds the twist it calls self-modifying: the model generates its own values, the targets its memory is trained toward. It is, quite literally, learning its own update algorithm as it reads — Schmidhuber's 1993 self-referential weight matrix, made trainable at scale.

The update rule underneath is the paper's Delta Gradient Descent — a delta-rule variant whose decay adapts to each input, chosen because token streams are highly correlated and a plain gradient step treats every sample as independent. Only the query projection stays fixed; everything else about how this memory learns is itself learned, in context.

figure 7 · the architecture

HOPE, assembled

Put the two together and you have HOPE: the self-modifying sequence model where working memory used to be, followed by a CMS chain where the MLP used to be. Side by side with a Transformer block, it's the same silhouette — a token mixer, then feed-forward blocks — but every part now has a clock, and the clocks cover the whole spectrum.

Two details worth keeping. The components are separable — a HOPE-Attention variant keeps ordinary softmax attention and just adds CMS, and still beats the Transformer, so the memory spectrum helps independently of the self-modification. And existing pre-trained Transformers can be adopted into the scheme: initialize CMS blocks from their trained MLPs and let the levels adapt from there — which is how the continual-learning experiments below start from Llama.

results · the scoreboard

Better as a language model — much better at remembering

As a plain language model, HOPE leads its weight class. Trained from scratch at 1.3B parameters on 100B tokens, against every strong baseline trained identically:

The same ordering holds at 760M, and the gap widens with scale. On long context, HOPE holds 100% on passkey retrieval at 16K where the Transformer manages 79.8%, and on BABILong it keeps answering out to 10M tokens — territory where GPT-4-class models have already collapsed by ~256K. In the continual learning tests it was built for — class-incremental classification, learning two new languages in sequence — HOPE-adapted Llama models beat in-context learning, EWC and external-learner baselines, and adding memory levels visibly closes the forgetting gap. The ablations are clean: remove the delta update, the momentum, the weight decay or the CMS, and quality drops every time.

the honest column attention is not beaten everywhere: on short in-context recall (SWDE, SQuAD and friends) Transformers still lead, and on multi-key retrieval softmax attention stays clearly ahead of every parametric memory, HOPE included. The paper says so plainly — and its own conclusion states that catastrophic forgetting is reduced, not solved.

the equations · worked

Memory, momentum, delta, clock

applied · our reading

Continual learning is the sovereign feature

• The gap it names is our gap. A domain assistant lives on a moving target — notifications, rate changes, case law. Today the options are stuffing context (expensive, forgotten at session end) or periodic retraining (slow, and it risks trampling what the model already knows). A memory spectrum that consolidates at run time is the first architectural answer to that problem, not an ops workaround.
• The reframe is useful even without the architecture. Seeing fine-tuning, in-context learning and pre-training as one mechanism at three speeds is a planning tool: it says choose the clock for each kind of knowledge — statutes slow, rates medium, the current case fast — instead of forcing everything into either the prompt or the weights.
• Watch the adoption path, not just the benchmark. The result that matters for us is the Llama adaptation: CMS blocks initialized from a pre-trained model's own MLPs, then continually trained. If that recipe matures, it upgrades models we already run rather than demanding a new pre-train.

the caveat we would flag these are proof-of-concept scales — 760M and 1.3B, tens of billions of tokens — and a stack of interacting parts (self-modification, five inner memories, a CMS schedule) that no one has yet run at production scale or hardened operationally. Recall-heavy lookups still favor attention, so a deployed system keeps retrieval regardless. This entry is a bet on a direction, not a migration plan.

And the direction rhymes with the whole arc. Entry 12 removed the human from the reward. Entry 13 removed the quadratic bill. This one aims at the wall between training and inference itself — if it falls, "shipping a model" stops meaning freezing one.

← back to the knowledge base — the full index, searchable by tag.