One nudge = one flashcard from the text: a word, and the word that really comes after it. The model bets on what comes next, checks the text, and moves all its numbers a tiny bit so the right word would get a bigger bet next time. Step size is how big that tiny bit is. New start throws every number back to random. Card size is how many numbers each word's cards hold: 2 draws a flat map, 3 a 3D one. Changing it starts training over, because the model itself changes size.
Keys: Space plays or pauses, N does one nudge.
Measured after each read of the text: 0 would mean it bet 100% on every right answer. It can never get there, because some words are followed by several different words, so even a perfect model has to split its bet. The red line is the best any one-word-memory model could ever do on this text.
Reads 0 · Nudges 0 · Start #7 · –
Every word is drawn at the numbers on its cards. A dot ● is a word's question card, the numbers it uses when it asks “what comes after me?”, so words followed by the same things end up together. A diamond ◇ is its answer card, used as a contestant, “pick me as the next word!”, so words that come after the same things end up together.
Click a dot to focus on that word: its question card, plus the answer cards of the words that really follow it. Click it again, or on empty space, to go back. Scroll to zoom; drag to move (in 3D, drag to turn it). The colours are just for you: one per word, never per group, so they can't give the answer away.
● shows or hides a word's question card, ◇ its answer card. Click a name to focus on that word, and click it again to go back.
The model never picks just one word. It spreads 100% of its bet across every word it knows. Pick a word to see how it bets on what comes next: the striped bars show how often each word really comes next in the text, the solid bars are the model's bet.
Press One nudge to see a single training step here.
| word | ● question card | ◇ answer card |
|---|
This is the whole model: the flashcards, the cards of numbers, the betting, the checking and the learning. Five short sections, and most of it is comments.
The page reads this code straight out of itself, so it's exactly the code that runs the Play tab. Everything else on the page only draws pictures of what this code is doing.
Picture a baby robot that has read one short text, over and over. It can't read letters. It only has one game: I show you a word, you guess the next one. Every word gets turned into a couple of numbers, and then it's the same loop that trains the big models: guess, check, blame, nudge. Here it is in seven steps.
Every pair of neighbouring words is one flashcard.
the fox has a hat gives four: the → fox, fox → has,
has → a, a → hat. The front is the asking word, the back is the
right answer, and the text supplies the answer for free. Before
each read, the deck gets shuffled so the order doesn't matter.
A question card ●, used when the word asks "what comes after me?", and an answer card ◇, used when it's a contestant, "pick me as the next word!". Each card holds 2 numbers (or 3 in 3D), so we can draw it as a dot. At the start, every number is random: the robot knows nothing. Real models use hundreds of numbers per card.
To guess what comes after fox, it takes fox's
question card and multiplies-and-adds it with every word's
answer card: first number × first number, plus second × second.
Cards pointing the same way score high. That gives one score per
word in the dictionary.
The scores go through softmax: every +1 of score makes a word's bet 2.7× bigger, and then the bets are scaled to add up to exactly 100%. That's the betting slip. Only the gaps between scores matter, so a word far ahead of the rest gets almost all of the bet.
Flip the card over. How wrong it was is
−log(bet on the right word): 0 if it bet everything
on the right word, and bigger the less it bet on it. The "how
wrong" number on the Play tab is the average over every flashcard
in the text.
For every word:
push = what it should have bet − what it did bet. The
right word should have had 100%, every other word 0%. So the right
word gets a positive push, and every word it wrongly bet on gets a
negative one. That tiny formula is what's left after the log and
the softmax cancel each other out.
The asking word's question card takes a small piece of every answer card: a piece of the right word's (positive push, so it moves closer) and minus a piece of the ones it wrongly bet on (so it moves away). Every answer card does the same with the asking word's question card. Step size decides how small the pieces are. Then the next flashcard, and again, thousands of times.
"You shall know a word by the company it keeps." (J. R. Firth, 1957)
This page uses friendly names. Here's what you'll see in articles instead:
| On this page | Grown-up name |
|---|---|
| flashcard | training example |
| question card ● | input embedding |
| answer card ◇ | output embedding (unembedding) |
| numbers per card (2D / 3D) | embedding dimension |
| every number in the model | parameters (weights) |
| multiply-and-add | dot product |
| score | logit |
| betting slip | softmax probabilities |
| how wrong | loss (cross-entropy) |
| push | gradient (with the sign flipped) |
| nudge | training step (stochastic gradient descent) |
| step size | learning rate |
| one read of the text | epoch |
| best possible | entropy of the next word, given one word |