Notes on machine learning
Technical notes written while learning: honest evaluation, data leakage, deployment, LLM applications and the mistakes behind each lesson. They come from the projects on this site, with the code one click away.
Papers, plainly: LoRA
The 2021 paper that made adapting giant models cheap fits in one idea: instead of touching every weight, learn a small low-rank shortcut and add it in. Second entry in the series, with a real experiment: a tiny GPT trained on Shakespeare and adapted to Don Quijote five different ways, to measure what each one actually costs and forgets.
Read article
I benchmarked my BPE tokenizer against tiktoken and BERT: the numbers were not kind
I retrained the BPE from transformer-from-scratch to an 8,000-token vocabulary and pitted it against cl100k_base, o200k_base, GPT-2 and multilingual BERT on the same English, Spanish and code corpora. The Spanish gap was worse than expected, and the real reason was not the one I had assumed.
Read articlePapers, plainly: Attention is all you need
The 2017 paper underneath every current chatbot fits in one idea: let each word decide which other words to look at. First entry in a series where I read classic papers and retell them the way I wish someone had told me.
Read article
When is it worth using an LLM — and when is it not?
LLMs solve certain problems better than anything else. Others they solve worse than a three-line classifier. The question to ask before opening the OpenAI API.
Read article