LoRA: Low-Rank Adaptation of Large Language Models
Instead of updating every weight during fine-tuning, LoRA freezes the base model and learns a low-rank update to selected weight matrices. The trainable parameter count drops by orders of magnitude, adapters can be swapped per task, and inference cost stays unchanged once the update is merged.
Key ideas
- Weight updates during fine-tuning have low intrinsic rank
- Two small matrices replace a full update, and merge back for free
- Which matrices to adapt matters more than the rank
Why read it now
It made fine-tuning large models practical on ordinary hardware and is the basis of most adapter ecosystems today.
Question to keep in mind
Why does merging the low-rank update into the base weights add no inference latency, unlike adapter layers?