The winning subnetwork was in there all along

figure 1 · the whole idea

A smaller network was in there all along

Train a big neural network and most of its weights turn out to be nearly useless — you can delete 80–90% of them and barely dent its accuracy. That much was already known; it's called pruning. The puzzle was that the pruned small network, if you tried to train it from scratch on its own, learned badly. So the smallness seemed to be a property you could only reach by training big first.

This paper found the loophole. Inside the big network there is a small subnetwork — a winning ticket — that trains just fine on its own, reaching the full network's accuracy in about the same number of steps. The catch is that you have to keep its original starting weights. Same connections, same initial values, trained alone.

click any figure to zoom · scroll to magnify · drag to pan

The distinction that makes it surprising: it isn't that a small network can be good — we knew that. It's that the small network was already present, already well-positioned, at initialization, before a single step of training. Training didn't build it. Training revealed which one it was.

figure 2 · the metaphor

Why it's called a lottery

Here is the mental model the name comes from. A big network doesn't contain one subnetwork — it contains a combinatorially huge number of them, overlapping and sharing weights. Think of each as a ticket. Every ticket starts with whatever random initial values it happened to be dealt.

Training is the draw. A few tickets began with an initialization that happens to sit in a good spot — close enough to a solution that gradient descent can get there. Those are the winners. Most tickets are duds.

This reframes something everyone had noticed but couldn't explain: bigger networks are easier to train. Not because the extra capacity is used — most of it isn't — but because buying more tickets makes it likelier that at least one of them is a winner. Over-parameterization is buying lottery tickets in bulk.

figure 3 · iterative magnitude pruning

Train, prune, rewind, repeat

The recipe is a loop. Train the full network. Delete the weights that ended up smallest — closest to zero, so presumably least important. Then, for the weights that survived, throw away their trained values and put back the exact numbers they started with. Retrain from there, and prune a little more. Round after round, the network gets sparser and the winning ticket emerges.

Two details do all the work. Doing it iteratively — a small fraction per round over many rounds — finds much smaller tickets than one aggressive cut. And the rewind step in the corner is the whole discovery: you reset survivors to their original initialization, not their trained values, and not fresh random ones. The next figure shows why that one choice is everything.

figure 4 · the toy example

Magnitude pruning, by hand

Take one layer's weights. Magnitude pruning ranks them by absolute value — how far from zero they are — and removes the smallest slice. A weight near zero barely affects the output, so it's the first to go. That's the entire rule; there is no cleverness to it.

Eight weights, cut the four nearest zero. The survivors — the tall bars — keep their positions and become the ticket's connections; everything below the line is set to zero permanently. Real networks prune gently, around 20% per round over many rounds, but each round is exactly this.

worth noticing Nothing here looks at what a weight does, only how big it is. That magnitude is a decent proxy for importance is itself a little surprising — and it's why the method is cheap enough to run many times over.

figure 5 · rewind vs. random

The ticket is a structure and an initialization

Here is the experiment that proves the point. Take a winning ticket's sparse structure and train it two ways. First, with its original initial weights — the rewind. Second, with the same connections but fresh random values. If the structure were what mattered, both would learn the same. They don't.

The rewound ticket climbs fast and lands at the full network's accuracy. Re-roll the initial weights and the identical structure learns slower and plateaus lower. So a winning ticket isn't a lucky shape — it's a lucky shape paired with the specific starting numbers it was dealt. Separate them and the magic is gone.

The paper found these tickets at 10–20% of the original size, matching or beating the full network and often training faster. The headline result — a small network that trains to full accuracy — had been thought impossible precisely because everyone had been re-initializing at random.

the equations · worked

Masks, the ticket condition, and the schedule

applied · our reading

The small model, found rather than trained

For the work we do — specialist models that have to run inside someone's boundary — this is the other half of the compression story. Distillation makes a new small model that imitates a big one; the lottery ticket says a usable small model may already be hiding inside the one you trained, if you can find it.

the caveat we would flag Finding the ticket isn't free — you must train the full network first, then prune and retrain over many rounds, so this buys a cheaper model to serve, not a cheaper one to build. And magnitude pruning leaves sparsity scattered unpredictably through the weights; without kernels or hardware built for it, a 90%-sparse model can run no faster than the dense one. The result is a profound statement about over-parameterization first, and a deployment recipe second.

next in the arc

Where this goes

Distillation copied a model down; the lottery ticket finds the small model already inside. The third entry in making models smaller takes yet another angle: LoRA doesn't shrink the model at all — it freezes it and trains a thin adapter beside it, on the theory that the change a fine-tune needs is far lower-rank than the model. Three papers, one question: how little is actually necessary?