- PYTORCH
- TRANSFORMERS
- NLP
Transformer from scratch
A GPT-style language model implemented from scratch in PyTorch: multi-head attention, causal masking and residual blocks written by hand, plus a byte-pair tokenizer and an ablation study. Code at github.com/delcenjo/transformer-from-scratch.
A GPT-style language model written from scratch in PyTorch: multi-head attention, causal masking, residual blocks and positional embeddings by hand, with basic operations instead of torch.nn.Transformer. The byte-pair tokenizer is handwritten too. The goal was to be able to look inside when something breaks.
- Val. perplexity6.89
- No pos. embeddings8.04
- Tokenizerhandwritten BPE
Writing it to understand it
Reading the attention paper is not the same as implementing the head reshape and getting the dimensions wrong three times. Every piece is written with basic operations and covered by tests, so the mechanics are visible and safe to poke at.
Ablations: breaking things on purpose
The character baseline reaches a validation cross-entropy of 1.93 (perplexity 6.89). Removing the positional embeddings pushes it to 8.04 — without positions, attention cannot tell token order apart. And the surprise: a single attention head scored 6.76, slightly better than several, at this model size.
What I keep
Now, when a transformer does not converge or a tokenizer splits a word in the wrong place, I have a mental model of which piece to look at. No course gave me that intuition: it came from implementing every tensor and watching it fail.
Interactive notebook
Runs real Python in your browser via Pyodide (WebAssembly), no server involved: the first load takes a few seconds to start the interpreter, and after that every cell can be re-run.
Real ablations and the trained model generating text in numpy
A runnable Jupyter notebook: real Python running in your browser via Pyodide, no server involved.
Next projects:
Credit Risk Platform
End-to-end MLOps platform for credit-default scoring: reproducible training with a versioned model registry, a FastAPI inference API with input validation, PSI drift monitoring and a Prometheus and Grafana observability stack, all containerised with CI. Code at github.com/delcenjo/credit-risk-platform.
This very site
This portfolio is hand-built with Next.js and deployed on Cloudflare Workers, with a model lab that runs entirely in the browser and continuous deployment on every commit. Code at github.com/jmweb-org/jmweb-pag-web.
Built with
