Calibrated probabilities: when a 0.7 has to mean 70%

You train a default classifier, look at the ROC-AUC and it is fine. Then someone asks what it means when the model says 0.7 for a specific client, and everything falls apart: in most freshly trained models, that 0.7 does not mean a real 70% probability of default. It means, at most, that this client scores higher than another at 0.5.

Ranking and probability are different things. ROC-AUC only measures order: whether positives score above negatives. A model can rank beautifully and be systematically off in magnitude — gradient boosting tends to produce probabilities that are too extreme, and regularised models too lukewarm. As long as you only rank (show the 100 riskiest clients), it does not matter. The moment someone decides with the number (approve below 0.2), it matters enormously.

How you check it: the reliability curve. Group the predictions into buckets (0-0.1, 0.1-0.2...) and compare each bucket's mean predicted probability with the observed real frequency. If the model says 0.3 and 30% of that bucket defaults, the curve sits on the diagonal. The Brier score summarises the same thing in one number: the lower, the better calibrated and more discriminant at once.

How you fix it. With a calibrator trained on data the model never saw: Platt scaling (a sigmoid over the score) when data is scarce, isotonic regression when there is enough. In scikit-learn, CalibratedClassifierCV does both. The detail that took me a while to understand: the calibrator needs its own data — calibrating on the same set the model was trained on lies all over again.

And only then, the threshold. With trustworthy probabilities, choosing the decision threshold stops being guesswork: you can compute the expected cost of every threshold with the real costs of each error type and keep the minimum. In my credit risk model that threshold came out at 0.14, nowhere near the default 0.5, because an unforgiven default costs five times more than a rejected good client.

The short lesson: before letting anyone decide with your model's probability, draw the reliability curve. Two hours of work that turn a decorative score into a number you can operate with.