The transformer, no smoke and mirrors

Before 2017, to process a sentence, models read it word by word, in order, dragging along a running summary updated with every new word. It is like trying to remember a long list someone reads out loud: by the time it ends, the beginning has blurred. Long sentences with distant dependencies got lost along the way, and reading in strict order cannot be split across processors either: training was slow.

The idea that changed everything: what if every word could look directly at every other word and decide which ones matter, instead of reading in a row? That is attention. To predict what comes after a word, the mechanism computes a score between that word and every other word in the sentence, and builds its representation by mixing the top scorers. No dragged summary, no distances: word 1 and word 500 end up equally close.

Under the hood, the trick has three pieces with odd names — query, key, value — but simple logic. Every word emits a question (query), wears a label (key) and carries some content (value). The score between two words is how well one's question matches the other's label, and what finally gets mixed is the content of the best-matching words. It is all matrix multiplications — exactly what a graphics card does absurdly fast and in parallel, so on top of working better, it trains far faster than reading in a row.

In the attention demo this stops being theory: these are the real weights, extracted from the trained checkpoint of a small transformer written from scratch, not an illustrative drawing. Pick a layer and a head, and watch exactly which earlier characters the model attends to when deciding which one comes next. You can see, character by character, the same query-key-value mechanics you just read about.

A single attention tends to fixate on one kind of relationship, so the transformer runs several in parallel — heads — and each one learns on its own to look at different things: one ends up tracking syntax, another neighbouring characters, another stranger relations. Nobody assigns the roles; they emerge from training. In the 3D attention view that same transformer's four layers and four heads are stacked like floors of a building you can orbit: pick a character and see at once what every head on every floor is looking at — the lower floors watch immediate neighbours, the upper ones the whole structure of the sentence.

Stacking many attention layers — each looking at what the previous one built — produces increasingly rich representations, and generating text needs one extra rule: the causal mask. When predicting the word at position 10, the model may only look at positions 1 through 9, never anything after — if it could see the future, it would learn to copy it instead of predicting it. With that restriction, generating text becomes repeating a single step: given the words so far, compute a probability for every possible next word, pick one, append it, and repeat.

Choosing which of those likely words gets appended at each step is not one strategy but several, and each has its own character. Greedy decoding always takes the most probable one; it is fast and deterministic, but can get stuck in flat, repetitive sentences by never taking a risk. Beam search keeps several candidate sentences at once instead of just one, and at the end picks the one with the highest joint probability — better quality, more computation. Temperature sampling adds controlled randomness: low temperature behaves like greedy, high temperature produces more surprising text and, past a certain point, delirious text.

In the beam search demo all three strategies run on the same character transformer trained on Shakespeare, with the decision tree and the dying branches on display. Push the sampling temperature until the text stops making sense, and compare that branch with how disciplined — and sometimes dull — beam search turns out on the very same model.

Everything in this lesson and the previous two — tokenizer, embeddings, attention, generation — comes together in train your own mini-GPT, the lab's capstone: you pick a text corpus, train your own BPE tokenizer on it, decide how many layers and heads your transformer will have, and watch it genuinely train, step by step, all the way to generating text with your own model. It is the same machinery this lesson took apart piece by piece, now assembled whole and running start to finish.