LoRA & Fine-Tuning

A 7B-parameter model is relatively small. Full fine-tuning needs ~56.0 GB just for optimizer states and gradients — that's already a serious GPU.

With rank 1, we capture only the dominant patterns in ΔW. The approximation is coarse (30%) but uses very few parameters — just 1 pairs of vectors.

With rank 4, LoRA uses only 4.1K parameters instead of 262K — a 98.4% reduction. Matrix B is 512×4 and A is 4×512; their product reconstructs the 512×512 update.

Applying LoRA to the Q (Query) projection in attention. Research shows Q and V adapters give the best results for most tasks, but adding K helps for complex reasoning.

Before training, B is initialized to zero and A to random small values, so B×A = 0 — the model starts identical to the original. Only B and A will receive gradients; W₀ stays frozen.

At α = 1.00, you get the standard LoRA merge. The key advantage: you can hot-swap lightweight adapters (~MBs) for different tasks while keeping one shared base model (~GBs).