Judgement · Machine learning · Madrid
When does AI actually make sense?
I study machine learning in Madrid and one of the things that interests me most is judgement: knowing when a problem deserves a model, when a simple rule is enough and when an LLM is money spent for nothing. This page sums up what I have learned evaluating my own projects.
Before training a model
Most ML projects fail before they start: a poorly defined goal, insufficient data or a metric that does not measure what matters. I have run into this in my own projects, which is why I now always start with the question and the data, not the algorithm.
Reviewing models, starting with mine
I review my models for data leakage, optimistic metrics and badly chosen thresholds. Several of my open-source tools came out of that practice: splitcheck to detect leakage between train and test, and evalgate to block a degrading evaluation in CI.
The baseline first
Before trying anything sophisticated, a simple rule or a linear model sets the bar. If the complex model does not beat the baseline by a margin that matters, it does not deserve the complexity. It is the lesson I have had to relearn the most times.
No hype
Sometimes the answer is that you do not need an LLM, or that the simplest model wins. I would rather document that in the README, even if it makes the project less flashy, because judgement is trained the same way models are: on honest examples.
Want to talk judgement, data or an opportunity?
Everything I claim here can be checked in the code. If you want to discuss a case, give me feedback or tell me about an opportunity, write to me.
Write to meFAQ
When is machine learning worth it?
When there is a decision that repeats many times, historical data that contains the signal to make it better, and a clear metric for what improving means. If any of the three is missing, a model does not fix the problem: it hides it.
And when is an LLM worth it?
When the task is about language and genuinely needs understanding or generation: summarising, extracting, answering over documents. For classifying text into fixed categories, a classic classifier trained on good examples is usually cheaper and more stable. In llm-eval-harness I compare both with numbers, and I wrote up the conclusion on the blog.
How do you spot an inflated evaluation?
The usual signs: metrics too good for the problem, preprocessing fit outside the pipeline (the classic leak), a test set too similar to train (duplicates or temporal neighbours), and thresholds chosen by looking at the test set itself. My tools splitcheck and dsdiff were born to catch exactly that.
Where can I see these criteria applied?
In every project at github.com/delcenjo: the READMEs document what worked, what did not and why. And on this site's blog I write up the general lessons: data leakage, real regression or noise in an eval, when an LLM is worth it.
Can I run a case by you or ask your opinion?
You can, with one honest condition: I am a student, not a consultant. I will tell you what I know and what I do not. If you want technical conversation, mutual feedback or a junior with this kind of judgement for your team, write to me through the form.