The glossary

The percentage of predictions the model gets right overall; misleading when one class is far more common than the other.

The de facto default optimizer in deep learning: it adjusts the learning rate of every weight separately using the history of its gradients, so it converges well with barely any tuning.

The classic recommender-system algorithm: it decomposes the ratings matrix into two smaller matrices (user tastes, film traits) and solves them by alternating one and the other with a closed-form regression.

The first honest step of any data project: looking at types, nulls, distributions and correlations before training anything. Half of a model's problems are visible right here if you stop to look.

Finding structure in data that carries no labels at all: grouping similar points, reducing dimensions, spotting what is unusual. Nobody tells the model what to look for.

The branch of machine learning where an agent learns by trying: it acts, gets a reward (or a punishment), and adjusts its behaviour to earn more of the first and less of the second.

Training on examples that already come with the correct answer attached, so the model learns to predict it on new cases; most of what runs in production today is this.

A model that learns through a series of yes-or-no questions about the data, each one picking the split that best separates the classes; easy to read, but with enough depth it memorises instead of generalising.

The trick that lets a model decide, for each word, which other words in the text deserve more weight.

Instead of computing a single attention, a transformer computes several in parallel, each free to learn a different kind of relationship between words; they all get combined at the end.

Manufacturing variants of the examples you already have — rotating an image, adding noise, rephrasing a sentence — so the model sees more variety without needing genuinely new data.

A network that learns to compress every example down to a handful of numbers (the latent space) and then rebuild it from there; if the reconstruction comes out well, those few numbers were capturing what mattered.

The algorithm that spreads the blame for an error across every weight in a network, layer by layer going backward, using the chain rule; it is what makes training networks with millions of parameters possible.

The simplest possible model (sometimes just guessing the most common class), used as the minimum bar anything fancier has to beat.

A way of generating text that, instead of keeping only the single most likely word at each step, tracks several candidate sentences at once and drops the worst ones; it produces somewhat better text than greedy decoding at the cost of more computation.

The classic ranking algorithm of information retrieval: it scores each document by combining how rare each query word is, how often it appears (with saturation), and how long the document is. It is still hard to beat for keyword search.

Resampling the data you already have, with replacement, to simulate what a different sample from the same population might look like; it is how you estimate a number's uncertainty without needing more data.

The algorithm almost every language model uses to chop up text: it repeatedly merges the most frequent pair of symbols until it reaches the vocabulary size you asked for. Rare words end up split into pieces while common ones stay whole.

Adjusting a model so that when it says '80% probability,' that outcome really happens about 80% of the time.

Keeping part of a pretrained model's weights fixed during fine-tuning, and training only the rest; it is what makes transfer learning with very few examples possible.

A number between -1 and 1 that measures how much two variables move together; useful for spotting redundant columns, but on its own it never proves that one causes the other.

Two variables can move together without either causing the other: there might be a shared third cause, or plain coincidence. Confusing the two is one of the easiest ways to draw the wrong conclusion from a perfectly correct analysis.

Rounding an already-trained model's weights down to fewer bits — from float32 to int8, even int4 — so it takes up less space and computes faster; going from fp32 to int8 the accuracy loss is usually almost invisible.

How training and validation error change as the amount of data or training steps grows; reading it well tells you whether to get more data, add more capacity, or simply stop.

When one class shows up far more often than another in the data, like fraud versus normal transactions; that is where accuracy lies, and you need to look at recall, precision or PR-AUC instead.

The method almost all of machine learning uses to learn: work out which direction makes the error worse the least, take a small step the other way, and repeat.

Training a small model (the student) to mimic the outputs of a larger, already-trained one (the teacher), instead of learning directly from the original labels; it usually ends up more accurate than training that same small model from scratch.

How far, on average, the data strays from its mean, in the same units as the original data; it is the square root of the variance, which is why it reads more naturally.

The bell curve that turns up everywhere in statistics, because adding up many small, independent things tends to produce it; mean and standard deviation describe it completely.

The phenomenon, described around 2019, where a model's test error drops, spikes exactly when its parameter count matches the amount of data, and then drops again beyond that point; it broke the classic intuition that more parameters always means worse.

When new data stops resembling what the model saw during training; the model keeps giving answers, but gets them wrong more often.

During training, it randomly switches off a fraction of neurons at every step, so the network cannot lean too hard on any single one. It is one of the cheapest ways to fight overfitting.

Turning words, sentences, or images into lists of numbers so similar things end up close together in that numeric space.

One full pass through the entire training set; a model usually needs dozens or hundreds of epochs before it stops improving.

The compressed numeric space where a generative model keeps its representation of the data; moving smoothly through it usually produces in-between examples that exist in no dataset.

Turning a raw signal (audio, text, an image) into numbers a model can use; for example, running audio through an FFT and keeping the energy in a handful of frequency bands instead of the whole sound.

A single number that balances precision and recall, handy when you care about both and don't want to pick one.

A central store where the variables that feed models get computed and versioned, so training and production both use the exact same definition of each one. Without it, the two worlds quietly drift apart without anyone noticing.

Continuing to train an already-built model on your own data, to specialize it for one task without starting from scratch.

When information from the future or the target itself sneaks into training by accident; the model looks brilliant and then fails in production.

The number a model tries to minimize during training; it measures how wrong it is on the examples it sees, and its exact shape (squared error, cross-entropy...) changes which mistakes get punished hardest.

Two networks competing: a generator trying to fabricate believable fakes, and a discriminator trying to catch it. Trained together, the generator ends up producing things that are surprisingly convincing.

In very deep networks, the gradient travelling backward through backpropagation can shrink to almost nothing (and the early layers stop learning) or grow out of control (and the weights blow up). Layernorm and residual connections exist largely to prevent this.

Any setting of the model or the training process that is decided before it starts and is not learned from data: the learning rate, the number of layers, a tree's maximum depth.

A range that probably contains the true value, instead of a single number pretending to certainty; it widens when data is scarce and narrows when it is plentiful.

The most famous clustering algorithm: assign every point to its nearest centroid, move each centroid to the centre of its points, repeat until nothing shifts. You have to tell it upfront how many groups to look for.

How long a model takes to respond once it is trained; it sometimes matters more than accuracy, because a perfect model that takes three seconds may be useless for a real-time application.

Rescales the activations inside each layer to zero mean and unit variance, example by example; it stabilises training in deep networks, and transformers carry it in almost every corner.

A language model trained on enormous amounts of text with a huge parameter count; it predicts the next word so well that, at that scale, it starts to look like reasoning.

A table that crosses what the model predicted against what was actually true, cell by cell; precision, recall and every classification metric fall out of it directly, and it is the most honest way to see exactly where a model gets it wrong.

The part of a machine learning project that isn't the model itself: versioning data and code, deploying it, and watching that it keeps working over time.

Models that learn to undo noise: they are trained by wrecking images little by little down to pure static, and then learning the reverse process. Generating means running that reverse process starting from pure noise.

Instead of always picking the most likely word, the next one is drawn at random according to the candidates' probabilities; temperature decides how much: low, and the top candidate almost always wins; high, and the text turns erratic.

Putting every input variable on a similar scale before training; without it, a variable measured in the millions drowns out one measured in single units, and some models learn worse or slower because of it.

Watching production not just for whether the service responds, but for whether the model's predictions still make sense: input distribution, output distribution, metrics whenever labels are available.

Turning a category (red, green, blue) into a vector of zeros with a single one at that category's position; it stops a model from inventing an order between categories that never had one.

When a model memorizes its training data instead of learning the general pattern, and so performs worse on data it hasn't seen.

The probability of seeing a result as extreme as yours if there were actually no effect at all; a low p-value suggests what you see is not pure chance, but it says nothing about how large or important it is.

The oldest artificial neuron, from 1958: weights, a sum and a threshold. It converges if the classes can be split by a straight line; if not (like with XOR), the line dances forever and never settles.

A measure of how 'surprised' a language model is by a piece of text; the lower it is, the better it predicts what comes next.

The chain of automated steps that carries data from its source to the final prediction, with no manual steps in between.

Switching off a trained model's smallest weights, sorted by magnitude, because there almost always are too many: trained networks are enormously redundant, and you can prune a large share before accuracy takes a real hit.

Instead of learning the value of every action, this reinforcement learning approach adjusts the policy — the rule that decides what to do — directly, nudging it toward the actions that ended up paying off.

Like ROC-AUC but focused on catching the rare positive cases; more useful when what you're looking for is a minority, like fraud.

Of everything the model flagged as positive, how much actually was; high precision means few false alarms.

A number that measures how much a variable's distribution has shifted between two periods; a big jump means something has really changed.

A reinforcement learning algorithm that learns the expected value of every action in every state, purely from rewards and punishments, with no need for a model of the environment. Punishment is learned in one update; reward has to propagate step by step.

Before answering, the model looks up relevant information in a set of documents and leans on it, instead of relying only on memory.

Many decision trees, each trained on a different resample of the same data, voting together; a single tree is unstable, but the average of fifty is usually surprisingly solid.

Of all the positive cases that actually existed, how many the model caught; high recall means few slip through.

A network that learns small filters which slide across an image hunting for local patterns — edges, textures — and combines them across layers into progressively more complex shapes. It was the dominant vision architecture before transformers.

Many artificial neurons wired in layers, each one a weighted sum followed by a nonlinearity; stacked and trained on data, they approximate functions nobody could hand-code.

A network that processes sequences step by step, carrying a running summary (the hidden state) that updates at every element; transformers displaced it almost entirely because that summary blurs out over long sequences.

Training a model again on more recent data once its performance starts to slip; it can run on a schedule, or trigger automatically once drift crosses a threshold.

The place where trained models get versioned along with their metrics, their source data and which version is currently serving in production; without it, knowing which model produced a given prediction is an act of faith.

The simplest model for predicting a number: a weighted sum of the input variables plus a constant. It works as an honest baseline before reaching for heavier artillery.

Like linear regression but for classification: it squashes the weighted sum through a function that maps it between 0 and 1, and that number reads as a probability. Simple, fast and surprisingly hard to beat on plenty of problems.

Any technique that deliberately penalises a model's complexity so it does not memorise the training data; L1 and L2 do it by adding the size of the weights to the error being minimised.

The most common activation function in modern networks: it passes positive numbers through unchanged and zeroes out the negative ones. It is absurdly simple and outperforms fancier alternatives.

Splitting the data into three chunks that never mix: one to train on, one for adjusting decisions along the way (validation), and one touched exactly once, at the end, to actually measure performance.

The property that lets someone else (or you, six months from now) rerun an experiment and get the same result; it demands fixed random seeds, versioned data and code, and a recorded exact environment.

A score from 0 to 1 for how well the model tells two classes apart, averaged across every possible threshold; 0.5 is a coin flip, 1 is perfect.

In the bias-variance tradeoff, the part of the error that comes from the model being too simple for the problem; more bias means less overfitting but also less ability to capture the real pattern.

When the data you collected does not represent the real population because the very way you gathered it already excluded part of it; a model trained on it generalises poorly even when every internal metric looks perfect.

That a result is unlikely to be pure noise, judged against a threshold you set beforehand; significant is not a synonym for important, and with enough data even a tiny difference can clear the bar.

Measures how much two vectors point in the same direction, ignoring their size; it is the standard way to compare two embeddings, ranging from -1 (opposite) to 1 (identical direction).

Two ways to fight class imbalance before training: oversampling repeats (or synthesises) examples of the rare class, undersampling trims examples of the common one. Neither one creates genuinely new information out of thin air.

Turns any list of numbers into probabilities that add up to one, exaggerating the differences: the highest number takes almost all the weight. It is the typical final layer of a multi-class classifier.

The opposite mistake to overfitting: the model is too simple to capture the real pattern, so it does poorly on both training and new data. The fix is almost never more data; it is more capacity.

It looks for the boundary that separates two classes while leaving the widest possible margin on both sides; with the kernel trick it can bend that boundary without ever explicitly computing in higher dimensions.

How many examples a model sees before updating its weights all at once; larger batches give steadier gradients but are slower to compute per step, smaller ones are noisier but sometimes generalise better.

How far a model moves at each training step; too large and it bounces around without settling, too small and it takes forever to learn anything.

The chunks a language model splits text into to process it; not always full words, and they're what determines how much using the model costs.

Starting from a model already trained on a large task and reusing what it learned for a new, smaller task instead of starting from scratch; with fifteen examples and a frozen model underneath, most real vision problems get solved this way.

The architecture behind most current language models; it processes the whole text at once instead of word by word.

A mathematical shortcut that lets a model behave as if the data lived in a space with many more dimensions, without ever computing those dimensions explicitly; it is how an SVM finds boundaries that would be impossible in the original space.

The cutoff point that turns a probability into a yes-or-no call; moving it trades off false positives against false negatives.

Choosing a model's cutoff not to maximise accuracy, but to minimise real-world cost: a false negative in fraud detection does not cost the same as a false positive, and the threshold should reflect that.

Training and testing the model several times on different slices of the data, so you're not trusting one lucky split.

A point so far from the rest that it demands an explanation: it might be a data-capture error, or it might be real and valuable information. A histogram or box plot usually gives it away instantly.

The average of the squared distances between each data point and the mean; it measures spread, and in machine learning it also names how much a model's predictions swing if you retrain it on slightly different data.

How much text a language model can see at once before it has to forget what came earlier; anything outside that window simply does not exist for it at that moment.

Keeping a history of exactly which version of the data trained each model, the same way code gets versioned; without it, reproducing a result from six months ago can be impossible.

The part of machine learning that works with images and video: recognising objects, detecting faces, tracking motion. Almost all of it today runs through networks trained on millions of labelled images.

A 2013 model that learns word embeddings purely from which words appear together, with nobody telling it what anything means; the classic result is that king - man + woman lands near queen.