1Diffusion Models 2Retrieval-Augmented Generation 3LoRA & Parameter-Efficient Fine-Tuning 4RLHF

Lensa ML

LoRA & Fine-Tuning

Step 1 of 6: The Fine-Tuning Problem

Why updating all parameters is expensive

Model size (billions)7

Total params

Weights memory

28.0 GB

Training memory (16 B/param)

112 GB

A 7B-parameter model is relatively small, yet Adam's two optimizer states alone take ~56.0 GB. That already needs a serious GPU.

Drag Model size from 1 to 70 billion and watch the memory readouts grow. The weights readout assumes 32-bit floats, 4 bytes per parameter. Full fine-tuning with Adam needs more: a gradient for every weight plus two moment estimates, about 16 bytes per parameter in fp32 before activations. For a 70B model that is roughly 1.1 TB. Mixed precision lowers the total without changing the pattern. Full fine-tuning also leaves a complete copy of the model for each task, and that storage cost was as much a target for LoRA as memory.

Step 2 of 6: Low-Rank Intuition

Weight updates are low-rank

Rank r1

Rank

Factor params

Full ΔW params

With rank 1, ΔW is built from just 1 pair of vectors: 16 numbers instead of 64. Only one broad pattern can be represented.

Drag Rank r from 1 to 8 and watch the 8 × 8 result fill in as rank-1 terms are added. Each term is an outer product u vᵀ, a column times a row, which builds a full 8 × 8 matrix from 16 numbers. A sum of r such terms has rank at most r. Hu et al. checked this in GPT-3: updates learned at rank 8 and at rank 64 shared their top directions, and r = 1 already did well on some tasks.

Step 3 of 6: LoRA Decomposition

ΔW ≈ B × A

Rank r4

Full params

262K

LoRA params

4.1K

% of full

1.56%

Rank

With rank 4, LoRA uses only 4.1K parameters instead of 262K, a 98.4% reduction. Matrix B is 512×4 and A is 4×512; their product reconstructs the 512×512 update.

Drag Rank r from 1 to 32 and compare the full and LoRA parameter counts. For a d_out × d_in weight, B is d_out × r and A is r × d_in, so LoRA trains r(d_in + d_out) numbers instead of d_in · d_out. With d = 512 and r = 4 that is 4,096 against 262,144. The forward pass computes h = W₀x + (α / r) · B(Ax), applying A first so ΔW is never built in training. Dividing by r reduces retuning when r changes. On GPT-3 175B, LoRA cut trainable parameters by about 10,000 times.

Step 4 of 6: Where to Apply LoRA

Choosing transformer components

Component: Q (Query)0

Component

Full params

262K

LoRA params

8.2K

Applying LoRA to the Q (Query) projection in attention. On a fixed parameter budget, the original LoRA paper did best by adapting Q and V together.

Move the Component slider across Q, K, V, O, and FFN and compare the parameter counts at rank 8. A square 512 × 512 projection costs 8,192 LoRA parameters, while the 512 → 2,048 feed-forward layer costs 20,480. The original paper adapted only attention weights in GPT-3 and left the feed-forward layers frozen. QLoRA (Dettmers et al., 2023) later found that putting adapters on every linear layer mattered more than the choice of rank. It fine-tuned a 65B model on a single 48 GB GPU by storing the frozen base in 4-bit precision.

Step 5 of 6: Training LoRA

Frozen weights, trainable adapters

Training step0

Step

Accuracy (illustrative)

30.0%

W₀ status

Frozen

Before training, B is zero and A holds small random values, so B×A = 0 and the model starts identical to the original.

Drag Training step from 0 to 20 and watch B and A update while W₀ stays frozen. Starting B at zero means the first forward pass is exactly the pretrained model, so training begins from a known good point. Only B and A need gradients and optimizer states, which is where the memory saving comes from. Activations still have to be stored, because backprop passes through the frozen layers to reach the adapters. LoRA runs often use a larger learning rate than full fine-tuning. For GPT-3, a LoRA checkpoint took about 35 MB instead of 350 GB.

Step 6 of 6: Merging & Stacking

Combining and swapping adapters

Adapter strength λ1.00

Strength (λ)

1.00

Performance (illustrative)

93%

Adapters

3 tasks

At λ = 1.00, this is exactly the standard LoRA merge, W₀ + (α/r)·BA. Here α is not a merge knob: it is a training hyperparameter (often 16 or 32) that scales BA by α/r, so changing r does not force a new learning rate. After merging, inference costs nothing extra, and lightweight adapters (~MBs) can be swapped per task on one shared base model (~GBs).

Drag the adapter strength λ from 0 to 2 and watch the LoRA bar and the performance readout. Merging computes W = W₀ + λ · (α / r) · BA once and stores the result, so inference costs the same as the base model. Older adapter layers add extra computation to every forward pass instead. Keeping adapters separate lets one server hold the base model and switch adapters per request. Adapters for different tasks can also be summed with their own weights, though they interfere when they change the same directions. Image tools use this λ to set a style LoRA's strength.

What is LoRA (low-rank adaptation)?

LoRA (low-rank adaptation) is a fine-tuning method that freezes a pretrained model's weights and trains a small low-rank update next to selected weight matrices. Each update is the product of two thin matrices, B and A, so a layer computes W₀x + BAx. Only B and A are trained, which is usually well under 1% of the model's parameters.

LoRA is the most common way to adapt large language models and image generators on modest hardware. It is used to fine-tune open models such as Llama and Mistral for chat, coding, or a company's own data, and to train style and character adapters for Stable Diffusion. Because each adapter is a few megabytes, one base model can serve many tasks, and the adapter can be merged into the weights so inference runs at the original speed.

Key ideas

LoRA freezes the pretrained weights and trains a low-rank update ΔW = BA beside each chosen matrix.
For a d × d weight, LoRA trains 2dr parameters instead of d², a large saving when r is small.
B starts at zero, so training begins from the exact pretrained model.
A trained adapter can be merged into the weights, so inference costs nothing extra.
One base model can serve many tasks by swapping adapters of a few megabytes each.
QLoRA pairs LoRA with a 4-bit frozen base to fine-tune very large models on one GPU.

Short takes on this topic

How LoRA fine-tunes a 7B model by training 0.4% of a layer

Common questions

What rank should I use for LoRA?

Ranks from 4 to 64 are common, and 8 or 16 is a typical starting point. A harder task or a larger gap from the pretraining data can justify a higher rank. If the adapter underfits, raising the rank or adding more target layers usually helps more than training longer.

What does lora_alpha do?

The update is scaled by α / r before it is added to the frozen output. Changing α acts much like changing the adapter's learning rate. Common settings are α equal to r or twice r.

Is LoRA as good as full fine-tuning?

On the tasks in the original paper, including GLUE with RoBERTa and DeBERTa and several GPT-3 benchmarks, LoRA matched or beat full fine-tuning. Later studies found a gap on tasks that need a lot of new knowledge, such as continued training on code or math, where LoRA learns less but also forgets less of what the base model knew.

Can LoRA be used for image models?

Yes. In Stable Diffusion, LoRA adapters are usually trained on the attention layers of the denoising network to teach a style, a character, or an object from a few dozen images. The resulting files are small enough to share and stack, which is why they became a common format in image-generation communities.

Related explainers

Transfer Learning — Standing on the shoulders of pretrained giants
Singular Value Decomposition — Factor any matrix into rotation, scaling, and rotation — the Swiss army knife of linear algebra
The Transformer Architecture — A visual guide to 'Attention Is All You Need'
RLHF — Align language models with human preferences using reward models and policy optimization
How Self-Attention Works — Query, key, value — the core of self-attention
Linear Algebra — Vectors, matrices, and the language of data

References

Last updated October 25, 2026