Papers, plainly: LoRA
The 2021 paper that made adapting giant models cheap fits in one idea: instead of touching every weight, learn a small low-rank shortcut and add it in. Second entry in the series, with a real experiment: a tiny GPT trained on Shakespeare and adapted to Don Quijote five different ways, to measure what each one actually costs and forgets.
Second entry in the series where I read classic papers and retell them the way I wish someone had told me (the first was Attention is all you need). This time it is LoRA: Low-Rank Adaptation of Large Language Models, Hu et al., 2021. I implemented it from scratch, tested it with a real experiment instead of taking a paper's word for it, and all the code is at github.com/delcenjo/lora-from-scratch.
The problem it solves. Adapting a large model to a new task, the classic way, means touching every one of its weights: if the model has seven billion parameters, storing an adapted copy costs seven billion parameters, even if the new task is tiny. There is a second, less talked-about problem: touching every weight also overwrites whatever the model already knew. Adapting well to the new task and keeping what was learned pull in opposite directions.
The paper's idea. The starting observation is that the update fine-tuning actually contributes, even though it is represented as a matrix the same size as the original weights, tends to have low rank: almost all the information in that update fits in far fewer dimensions. So instead of learning that full update matrix, it is learned as the product of two very narrow matrices, B and A, with a rank r much smaller than the original dimensions. The original weights, W0, are frozen entirely. The adapted layer computes W0·x plus (alpha/r)·B·A·x, instead of W·x directly.
Why B starts at zero. A is initialized with small random values; B is initialized entirely to zero. That is not a minor implementation detail: with B at zero, the product B·A is zero at the first step, whatever A contains, so the adapted model starts as an exact copy of the pretrained one. Training moves away from that point gradually. If both matrices started with random noise, the adapted model would start broken and the first iterations would go into undoing that noise before learning anything useful.
The weight-merging trick. Once trained, B·A (with its scaling) can be added straight into W0, leaving a single weight matrix, exactly as fast at inference as the original one: the low-rank structure saves memory and compute during training, not during inference once merged. I checked this with a test: the output of the model with LoRA unmerged and the output after merging the weights agree to the sixth decimal place.
The experiment. A tiny character-level GPT, two layers, two heads, 128-dimensional embeddings, 128-character context, 438,016 parameters. I pretrained it for 2,500 steps on the usual tiny Shakespeare corpus (the one from Karpathy's tutorials) and then adapted it to Don Quijote -- public domain, in Spanish, with a visibly different alphabet -- for 800 steps, five different ways: full fine-tuning, LoRA at rank 1, 4 and 16, and fine-tuning only the output layer. All on an ordinary CPU, 21 minutes total for all six phases.
The table, with real numbers. Full fine-tuning adapts best to Quijote (validation loss 1.76) but is also the one that forgets Shakespeare worst afterward: loss there rises from 1.91 to 3.57, worse than a model that had never seen English at all. LoRA rank 16 adapts almost as well (2.29) while touching only 5.31% of the parameters; rank 4 with 1.38% reaches 2.40; rank 1, with 0.35%, barely moves (2.55). Fine-tuning only the output head, with 2.83% of the parameters, surprisingly matches LoRA rank 16 (2.24) -- with a model this shallow, much of going from "predict English" to "predict Spanish" lives in which characters the final layer scores highly, not in how attention moves information around.
Catastrophic forgetting, in numbers. This is the part I actually cared about. With LoRA's adapters switched on, Shakespeare loss gets worse as rank grows (2.52 at rank 1, up to 2.90 at rank 16) -- at first glance this looks like more adapter capacity means more forgetting. It does not: LoRA's original weights are never touched at any point in these runs. What is rising is the adapter itself, trained purely on Spanish, pulling predictions toward Quijote even when the input is English. Switching the adapter off -- no reloading, just a boolean -- brings Shakespeare loss back to 1.906, 1.914 and 1.909 depending on rank, against an original of 1.907: differences of thousandths, within the validation's sampling noise. The pretrained model is still there, untouched, one flag away. Full fine-tuning and head-only training have no equivalent flag -- their original weights are gone the moment training starts.
What I am keeping. That "B starts at zero" is not an implementation trick but the reason any of this works without breaking things on attachment, that merging weights is free at inference and only expensive to explain, and that LoRA's reversibility is not a promise from the paper's introduction but something you can measure with two evaluations and a boolean. That said, this is a model with hundreds of thousands of parameters, not billions, and two modestly sized corpora: which rank wins here should not be extrapolated to a real model without rerunning the experiment at that scale. The honest takeaway is the direction of the results, not the third decimal place. The code, the tests and the notebook with everything executed are in the repository.
