How it actually learns

Training a model is, at its core, repeating three steps a great many times: measure how wrong it is, work out which direction to nudge each parameter to be a little less wrong, and nudge it a little in that direction. The first piece — how wrong it is — is called the loss function: a single number summarising, for one example or a batch of them, how far the model's prediction sits from the correct answer. The higher it is, the worse; the entire point of training is making it go down.

The second piece is the gradient, and it is the central idea behind all of this. For every parameter, the gradient says whether raising it or lowering it would reduce the loss, and by how much. It is exactly like finding the lowest point of a valley blindfolded: you cannot see the whole valley, but you feel the slope under your feet and take a step downhill. Repeat that for every parameter — there can be thousands — and you have gradient descent. In the gradient descent demo you can watch three balls with different step sizes race to the bottom of a 3D loss surface; it is the same mechanics that trains any network, drawn as a landscape you can actually look at.

The size of that step is the learning rate, and it is one of the most delicate numbers in the whole of training. Too small, and the model takes forever to reach the bottom — or gets stuck before it arrives. Too large, and instead of descending the valley it overshoots, bounces off the opposite slope, and can end up higher than where it started: exactly what you saw break the neural network in the previous lesson when you pushed that control up. There is no universally correct value; there is a reasonable range for each problem, and finding it is usually trial and error.

That leaves the question that actually matters: how do you compute the gradient of thousands of parameters at once? The answer is backpropagation, and its ingredient is a high-school rule: the chain rule. A network can be drawn as a graph of simple chained operations — multiply, add, apply a nonlinearity — and the final error gets propagated backwards through that graph, layer by layer, multiplying local derivatives at every step. Nobody differentiates a function of thousands of parameters by hand; each tiny piece is differentiated separately and the results are chained together.

In the backpropagation demo you can see that graph with real numbers: you step the forward pass forward node by node, and then the gradient flows backwards with the chain rule written out at every step, so you can trace exactly where each number comes from. Apply the gradients a few times in a row and you will watch the loss drop in front of you — the smallest possible version of what happens thousands of times a second while training any real network.

This idea was not always around. In 1958 Rosenblatt proposed the perceptron: a single neuron that adjusts its weights every time it gets something wrong, with a mathematical guarantee — if the two classes can be separated by a straight line, the perceptron converges, always. The trouble arrived with XOR, a problem so simple it fits in four points, and that no straight line can separate. Rosenblatt's rule has no way to solve it: the line dances forever and never converges. That failure, published in 1969, froze a good part of neural network research for nearly twenty years. In the perceptron demo you can see both sides: clean convergence with separable classes, and endless dancing with XOR.

The fix came from stacking neurons and training them together with backpropagation — exactly the network you broke with the learning rate in the previous lesson. A single neuron can only draw a straight line; several neurons combined, with a nonlinearity between layers, can bend that line until it separates any XOR and almost anything else. The perceptron was not badly designed: it just lacked companions and a way to train them all at once. That way is exactly what you just watched working, node by node, in the backpropagation demo.