How LoRA fine-tunes a 7B model by training 0.4% of a layer

Fine-tuning means training a pretrained model further on new data. Training moves each parameter, a learned number, along its gradient, the direction that lowers the error, and the update rule keeps extra statistics for each one. For 7 billion parameters that memory adds up. LoRA (low-rank adaptation) freezes the pretrained weights and trains a small add-on beside them instead.

1. 1
   LLaMA 7B, a language model with 6.7 billion parameters, has a model dimension of 4096: each token is a 4096-number vector. Each of its 32 layers has four attention projections, square matrices that turn a token's vector into a query, key, value or output vector. The gray square is one of them, the query weight W₀, 4096 × 4096 or 16,777,216 numbers. LoRA freezes it.
2. 2
   Beside W₀, LoRA adds two thin matrices. B is 4096 × 8 and A is 8 × 4096, where 8 is the rank r, the inner width they share. Their product B · A is a 4096 × 4096 update of rank at most 8. The layer computes h = W₀x + (α / r) · B(Ax), where x is the input vector and α is a fixed scaling setting. B starts at zero, so training begins from the exact pretrained model.
3. 3
   Training updates only B and A: 2 · 4096 · 8 = 65,536 numbers, 0.39% of W₀. Their fills, drawn to scale, are 1/512 as thick as the square. Only they need gradients and update statistics, which is where the memory saving comes from. The LoRA paper adapted only query and value weights in most experiments. Doing that in all 32 layers trains 4,194,304 numbers, about 0.06% of the whole model.