Models in the real world

The previous four lessons built a model that works: it knows what features and labels are, knows how to adjust itself with gradients, knows how to turn text into numbers and knows how to attend to the right part of a sequence. This lesson is about what happens next, once that model has to stop living inside a training notebook and start actually responding — on a phone with little memory, to data it never saw, or to a question nothing in training resembles.

The first problem is size. A trained model stores every parameter as a 32-bit floating point number — four bytes. Rounded to a single-byte integer with a shared scale factor, that same parameter weighs a quarter as much, and for making predictions the precision lost is usually minimal. In the quantization demo you can measure live, over 300 real images, how much accuracy this lab's digit classifier loses going from fp32 to int8, then int4 and int2: from fp32 to int8 almost nothing changes; at int2 the model collapses. That idea — sacrificing precision nobody needs — is what lets huge models run on a phone.

The second problem is redundancy: not every parameter in a trained network matters equally. Magnitude pruning sorts the weights by absolute size and switches off the smallest ones, on the assumption that a near-zero weight barely participates in the prediction. In the pruning demo you can watch the weight matrix go dark pixel by pixel while accuracy, measured live, holds up surprisingly well — up to 80% of the weights switched off in this model — before collapsing all at once. Trained networks carry far more capacity than they actually use.

The third problem is subtler: a model that only gives a number is telling half the truth. A prediction with no uncertainty attached is indistinguishable from a confident guess; in a real problem — approving a loan, flagging a diagnosis, deciding whether a car should brake — that difference matters. In the uncertainty demo sixty models trained on different bootstrap samples of the same data jointly compute a band: it narrows where many nearby points exist and fans open across any gap the model has never seen anything like. Fill the gap with your own points and watch the band close live.

The first lesson left a simple picture of overfitting: a U, validation error dropping then climbing as model capacity grows. That is true, but it is not the whole truth. Around 2019, double descent was described: keep raising capacity well past the point where the model exactly memorises the training data, and test error drops again, a second time, beyond the spike. In the double descent demo 128 models of growing width get trained and their test error plotted live: it drops, spikes exactly where parameters match the data count, and falls again beyond that — that second drop is, literally, where today's large models live, with far more parameters than training examples.

Even a model that ranks well can lie with its numbers. A classifier can have a good ROC-AUC — telling positives from negatives well — and still say 0.9 where the real accuracy frequency for that group is 0.6: it ranks beautifully and is systematically off in magnitude. Checking this is straightforward: group the predictions into buckets and compare each bucket's mean probability with the observed real frequency — if they match, the reliability curve sits on the diagonal. Only once that check is done does picking a decision threshold someone will actually act on make any sense.

The most complete example of all of this together is a retrieval-augmented system — RAG —, and here you can see the whole thing from the inside. In the RAG demo a document gets chunked into fragments, each fragment turns into a vector with exactly the same embedding mechanism from lesson 3, your question searches for its nearest neighbours through that same geometry, and the answer gets assembled citing the specific fragments backing it up. None of this is a black box: every stage of the pipeline can be inspected on its own.

None of these techniques makes a model perfect; they make it honest about its limits. Quantizing and pruning say how much can be trimmed before anyone notices, and exactly where that stops being true. Measuring uncertainty says when the model has no idea, instead of pretending it does. Calibrating says whether a number can be trusted before anyone decides with it. That honesty — not raw accuracy — is the difference between a notebook project and something that can actually be put to work.

And that closes the loop: from a table of features and labels to a transformer generating text, by way of how it learns, how it reads and how it becomes small and honest enough for the real world. Every piece of this course is still alive in the full lab, with more experiments than fit in five lessons, in case you want to keep pulling the thread.