Deconstructing LoRA
Low-rank adaptation in 50 lines of PyTorch — adapt a pretrained MLP to a new task with 4.59% of the trainable parameters, matching full fine-tuning.
Part 1
The Low-Rank Decomposition
Why fine-tuning weight deltas are low-rank, the BA factorisation, why A=random and B=0 init preserves the base, the alpha/r scaling trick.
Part 2
Fifty Lines of PyTorch
LoRALinear wrapper, recursive injector, 5-line merge_lora for zero-overhead inference.
View Code on GitHub
Part 3
4.59% Beats Full Fine-Tune
Adapt a moons-trained MLP to a 45°-rotated task. LoRA rank-2 (1,544 params) matches full fine-tuning (33,666 params) at 99.1% accuracy.