Papers, plainly: Attention is all you need
The 2017 paper underneath every current chatbot fits in one idea: let each word decide which other words to look at. First entry in a series where I read classic papers and retell them the way I wish someone had told me.
There is a 2017 paper with a cocky title — Attention is all you need — sitting underneath practically everything we now call generative artificial intelligence. I read it slowly, implemented its core piece from scratch, and this is the explanation I wish I had found on day one.
The problem before it. To process a sentence, the models of the time read it word by word, in order, dragging along a summary that kept getting updated. Like trying to remember a shopping list someone dictates: by the time they reach the end, the beginning has blurred. Long sentences and distant dependencies — "the girl we saw yesterday at the station was my cousin" — got lost on the way. Worse, reading in order cannot be split across processors: training was slow.
The paper's idea. What if, instead of reading in order, every word could look directly at every other word and decide which ones matter to it? That is attention. For the word "was", the mechanism computes a score against every other word in the sentence — how much does "girl" help me? how much does "station"? — and builds its representation by mixing the top scorers. No dragged summary, no distances: word 1 and word 500 are equally close.
Questions, labels and contents. The internal trick has three pieces with odd names (query, key, value) but library logic: every word asks a question (query), wears a label (key) and carries some content (value). The score between two words is how well one's question matches the other's label, and what gets mixed is the content. It is all matrix multiplications, which is exactly what graphics cards do absurdly fast and in parallel. That is the second half of the invention: it does not just work better, it trains far faster.
Several heads looking at once. A single attention tends to fixate on one kind of relationship. The paper runs several in parallel — heads — and each one learns on its own to look at different things: one ends up tracking syntax, another neighbouring words, another stranger relations. Nobody assigns the roles: they emerge from training. In my attention demo you can watch the real heads of my small transformer doing exactly this.
What I am keeping. First, the winning architecture did not add complexity: it removed it (recurrence out, convolutions out, attention only). Second, the paper barely mentions generating text — it was a translation system; the use that made it famous came later, from others. And third, reading it is far less brutal than it looks from outside: eleven pages, half of them tables. If this series helps one more person lose their fear of papers, it has already paid off.
