LLMs · RAG · Agents · Madrid
LLM applications, without trusting the demo
I study machine learning and build language model applications as personal projects: RAG over documents, agents that query data and assistants with tools. What has taught me the most is not making them work, but evaluating them: knowing whether they give correct answers or just answers that sound right.
RAG: answers that cite their source
Corpus indexing, semantic retrieval with embeddings and answer generation that cites the fragment it came from. The goal is for every answer to be verifiable, not to sound good. With chunking for long documents and re-ranking for large corpora.
Agents with guardrails
Agents that translate natural language into SQL queries and run them read-only, with well-defined tools. Useful without being dangerous: bounded scope, no write access by default, and a log of what the agent does. That is how I built llm-sql-agent.
Evaluation, not just a demo
The problem with LLMs is that they generate text that always seems correct. In my projects I evaluate answers with automatic metrics (ROUGE, BERTScore, faithfulness) and compare against classic baselines before calling anything done. An honest benchmark says more than a demo.
Fine-tuning when the API is not enough
For very specific tasks where a base model falls short, I practise supervised fine-tuning with LoRA or QLoRA on HuggingFace models. With evaluation before and after, because if you do not measure it, you do not know whether it was worth it.
Want to see the code, or tell me something?
The LLM projects are on my GitHub with their evaluation included. If you have feedback, a question or an opportunity, write to me.
Write to meFAQ
When does RAG make sense instead of just an LLM?
When answers need to come from specific documents, not from the model's general knowledge. RAG is the difference between 'the model knows about this in general' and 'the model searches the documentation and cites where the answer comes from'. Whenever answers must be verifiable and updatable without retraining.
How do you evaluate whether a RAG system works well?
With a set of questions with expected answers, faithfulness metrics (the answer comes from the retrieved context) and relevance metrics (the retrieved context is the right one). Retrieval also needs separate evaluation: the system has to find the right fragments before generating. That is what I practise in rag-document-assistant and llm-eval-harness.
What models do I use?
It depends on the project. The OpenAI and Anthropic APIs for most experiments, and open HuggingFace models (Llama, Mistral) when I want to practise fine-tuning or run everything locally. Part of the learning is exactly that judgement: choosing the model by problem and cost, not by hype.
Is an LLM always the answer?
No, and it is one of the lessons that cost me the most. In llm-eval-harness I compare an LLM against baseline classifiers on the same task: sometimes the LLM wins clearly and sometimes a cheaper classic model matches it. I wrote about this on the blog, in 'When is an LLM worth it'.
Can I ask you about this or propose something?
Of course. If you are learning the same things, if you have feedback on any of my LLM projects, or if you are looking for a junior for your team, write to me through the form. I always reply.