From data to your first model

A machine learning model is, at its core, a function: it takes in some input numbers and returns other numbers. No understanding involved, no magic — a function, like the ones you wrote in your first programming class, just with a great many parameters and a way of adjusting them that the next lesson covers. Predicting a house's price from its floor area, its age and its neighbourhood is exactly that kind of function: a vector of three numbers goes in, one comes out.

Each input number is called a feature, and the number you want to predict is the label. In the house example, floor area, age and neighbourhood are features; price is the label. Turning a real-world problem into this table — rows of examples, columns of features, one final label column — is almost always the first real job of any machine learning project, and it usually takes longer than training the model itself.

Training means searching for the parameters that minimise the error over the examples the model sees. The more parameters a model has, the more complicated a boundary or curve it can draw — this is called its capacity. With few parameters, a model can only manage rough approximations; with many, it can end up memorising every single point you showed it, noise included. That second option sounds great until you show it a new point: since it memorised instead of generalising, it fails.

That has a name: overfitting. Picture fitting a polynomial to a cloud of slightly noisy points: with degree 1 you get a straight line that barely comes close; with degree 15, a curve that snakes to touch every single point, noise included, and goes wild anywhere outside them. In the lab's overfitting demo you can raise the degree yourself and watch two curves split apart: the training error keeps dropping, while the error on data the model never saw draws a U — it falls, bottoms out, and climbs again. Finding the bottom of that U is, honestly, a good part of the craft.

That is why no serious model gets evaluated on the data it trained on. A slice — validation — is held back that the model never sees while fitting, and is used only to check whether it generalises and to choose its capacity (the polynomial's degree, a tree's depth, a network's size). A third set, test, is kept aside and touched exactly once, at the end, because even choosing hyperparameters by repeatedly checking validation ends up leaking its information into the model. We take evaluation you cannot trust more seriously in the final lesson.

The same problem can be solved with very different families of model. A decision tree carves boundaries with an axe: at every node it asks a binary question about one feature ("floor area > 80?") and chops the space into rectangles, smaller each time. A neural network instead combines many simple neurons to bend a boundary continuously and smoothly. In the decision tree demo you can paint two classes of points and watch those rectangular cuts appear one by one; give it plenty of depth with few points and you will see it memorise instead of learn — the same overfitting from before, wearing a different face.

In the neural network demo the same problem gets solved by a tiny network genuinely training in front of you: draw your own points, try the spiral preset to make it hard, and watch the boundary curve until it separates the classes. There is a control there — the learning rate — you can push until the network loses its mind and stops converging. Exactly why that happens, and what that number means, is the first question of the next lesson.