ML Models · Rigorous evaluation · Madrid
How I train and evaluate machine learning models
I study machine learning in Madrid and train models in personal projects with one rule: the evaluation is in charge. Leakage-free pipeline, cross-validation, calibrated probabilities and a cost-based decision threshold. A model that passes an honest test is a model I can trust.
Leakage-free validation
Preprocessing (scaling, encoding, imputation) lives inside the pipeline, so it is fit only on training data in each cross-validation fold. The metric I get in validation is what I can expect on new data, without inflation.
Calibrated probabilities
If the model is going to make decisions based on probability, that probability has to be reliable. I calibrate the model (Platt scaling or isotonic) and check it with the Brier score and the reliability curve. A 0.7 has to really mean 70%.
The threshold matters as much as the model
The decision threshold is not 0.5 by default. In my credit risk project I choose it by minimising the cost of each type of error: how much a false negative costs versus a false positive. That adjustment changed the result more than switching algorithms.
Per-segment error analysis
Global metrics hide problems. I analyse error by relevant subgroups to understand where the model fails and why. It is the part of the project where I have learned the most, because the failures are almost never where I expected.
Want to review my work or tell me something?
Every model is on GitHub with its pipeline, evaluation and documented decisions. If you have feedback or an opportunity, write to me.
Write to meFAQ
What types of models do you build?
Mainly tabular data models: classification (default, churn) and regression. Also NLP with transformers for text classification, extraction and semantic search, and LLM applications. All as personal projects with the code at github.com/delcenjo.
What does it mean for a model to be well calibrated?
That when the model says '70% probability of default', default actually occurs in around 70% of those cases. Without calibration, scores are a ranking but not real probabilities, and making decisions with them is fooling yourself with numbers that look precise.
Why not always use a 0.5 threshold?
Because the cost of being wrong is not symmetric. In credit risk, a false negative (lending to someone who defaults) can cost several times more than a false positive (rejecting someone who would have paid). The optimal threshold reflects that asymmetry, not a convention. It was one of the first big lessons of my credit project.
How do I know your metrics are not inflated?
You can check: the code of every project is public, with the pipeline, the splits and the evaluation. I also built tools to police myself: splitcheck detects leakage and duplicates between train and test, and evalgate compares evaluations in CI with significance tests.
Can I write to you about this?
Of course. Feedback on the methodology, questions, ideas, or an internship or first junior role in data or ML. The contact form goes straight to my inbox and I always reply.