Text that turns into numbers

A neural network only knows how to multiply and add numbers; it does not read. So before any language model can do anything with a sentence, that sentence has to become numbers, and that conversion has two separate steps that are easy to conflate: first chopping the text into pieces — tokenising —, then turning each piece into a meaningful list of numbers — the embedding. This lesson covers both, in order.

The first step decides what counts as a unit. The simplest option would be splitting by whole words, but that breaks quickly: any word the model has not seen in its vocabulary — a proper name, a typo, a word in another language — is left out, with no possible representation. The approach that won is splitting into subwords: pieces smaller than a word but larger than a letter, chosen so frequent words stay in few pieces while rare ones can still be built letter by letter if needed.

The algorithm that learns that split is called BPE, byte pair encoding, and its logic is pure counting. It starts with the text split into loose characters. At every step, it finds the pair of adjacent symbols that appears together most often across the whole corpus and merges it into a new symbol — "e" and "s" become "es" if that pair is the most frequent. Repeat that thousands of times and the vocabulary grows piece by piece: first merges of common letters, then syllables, then whole words if they are frequent enough. In the BPE tokenizer demo you can paste any text and watch those merges happen one at a time, the vocabulary growing and the text compressing merge after merge — it is exactly the algorithm sitting underneath the tokenizer of my handwritten transformer.

With the text already split into tokens, every token needs to become a list of numbers — its embedding — and this is where the interesting part starts. Nobody hand-picks those numbers: they come from training a model over enormous amounts of text with a single signal, the distributional hypothesis — words that show up in similar company tend to mean similar things. "Dog" and "cat" share sentences, owners and vets; their vectors end up alike. "Washing machine" lives in different sentences, and its vector lands far away.

In the word2vec demo that training happens right in front of you: sixty-nine words start at random positions on the plane, and within under a minute of real training, animals, food, places and weather have sorted themselves apart on the map. Nobody told it what an animal was; the model only saw which words appeared surrounded by which, and that was enough.

With words turned into lists of numbers, comparing meaning becomes geometry: two words are alike if their vectors point in similar directions, something measured with a formula called cosine similarity. Suddenly you can sort, group and search by meaning with the same machinery used to compute any distance. In the embeddings map you can type any word and watch it land on a two-dimensional plane next to the ones the model considers similar — sometimes surprising you with where it falls.

This geometry has a trap worth knowing before trusting it blindly. Two sentences sharing an ambiguous word — "bank", which can mean a riverbank or a financial institution — can score as very similar overall even when they talk about different things, while a genuine paraphrase with no shared words can score lower than it deserves. In the sentence comparator you can watch that failure live: alongside the global similarity number there is a word-by-word matrix showing which token answered which, and that is where you catch the global summary lying to you.

With tokens and embeddings settled, there is finally something to multiply. What is missing is the piece that decides what to do with that sequence of vectors — how one word knows which others to pay attention to within the sentence — and that piece is the transformer, the star of the next lesson.