Lensa ML
Lensa ML

Knowledge Distillation

Step 1 of 6: Teacher and Student

A big model trains a small one for serving

Student layers22
Student width (d_model)2048
Each bar is one layer; bar width is the model widthtargetsTeacher: 7B chat model32 × 4096, 6.74B, 13.5 GBStudent22 × 2048, 1.24B, 2.5 GBBatch-1 decode limit at an illustrative 1,000 GB/s:teacher ≤ 74 tok/s, student ≤ 402 tok/s
Student parameters
1.24B
Student weights (fp16)
2.5 GB
Size vs teacher
18%
Decode speedup
5.4×

A 22-layer student of width 2048 has 1.24B parameters and 2.5 GB of fp16 weights, 5.4× less to read per token than the teacher. The question is how much of the teacher's quality it can keep.

Drag Student layers and Student width (d_model) and compare the parameter count. Width costs more than depth: attention and FFN weights grow with d², so halving the width cuts each layer to about a quarter. The count uses 4d² per layer for attention, 3·d·ffn for the gated FFN and 2·V·d for the embeddings, which reproduces the teacher's 6.74B. Pruning and quantization shrink a trained model in place. Distillation is the one method that trains a new, differently shaped model, and it combines with both.

Step 2 of 6: Dark Knowledge

What soft targets say that labels don't

Handwriting: a 2 drifting toward a 315% toward 3
Bar scale
input imageHard label: one class gets 100%Teacher soft targets at T = 1 (illustrative logits)<0.1%<0.1%97%1.1%<0.1%<0.1%<0.1%1.4%0.2%<0.1%0123456789
Hard label
"2"
Teacher p(2)
97%
Teacher p(3)
1.1%
Other 8 classes
1.8%

The hard label says only "2". The teacher also says this "2" looks 1.3× more like a 7 than a 3, and almost nothing like a 0 or a 4. Switch to Log to see those small values.

Drag the Handwriting slider from a clear 2 toward a 3, then switch Bar scale to Log to read the small probabilities. Hinton, Vinyals and Dean called this information dark knowledge. Soft targets with high entropy give more information per training example than a label, and their gradients vary less from one example to the next. In their MNIST test, a student that never saw a 3 during training still classified 98.6% of test 3s correctly once its bias for that class was raised, learning the digit from how other digits resembled it.

Step 3 of 6: Temperature

Softening the teacher's probabilities

Temperature T4.0
softmax(z / T) of the teacher's illustrative logits for a "2"T = 1T = 4.000.514.0%05.6%138%212%34.2%46.3%54.4%613%77.8%84.9%9Entropymax 3.32 bits
Entropy
2.84 bits
p(2)
38%
p(7)
13%
p(3)
12%

Dividing the logits by 4.0 lifts "7" to 13% and "3" to 12% while "2" stays on top. The entropy rises to 2.84 bits, so the targets now say which wrong answers are close.

Drag Temperature T from 1 to 20 and watch the filled bars pull away from the dashed T = 1 outlines. Dividing every logit by T keeps their order but shrinks the gaps between them, so the softmax spreads out. The student uses the same T during training and returns to T = 1 at inference. Hinton et al. found that students with 300 or more units per layer did about equally well at any T above 8, while a 30-unit student did best between 2.5 and 4, where the most negative logits matter less.

Step 4 of 6: The Distillation Loss

Soft KL plus hard cross-entropy, and the T² factor

Weight on the soft term α0.70
Temperature T4.0
teacher at Tstudent at T (illustrative logits)0123456789Soft-term gradient size vs T (α = 1, log scale)1e-31e-21e-11e0with T²without T²15101520T
Soft term α·T²·KL
0.860
Hard term (1 − α)·CE
0.077
Total loss
0.937
Soft gradient size
0.459

KL at T = 4.0 is only 0.0767 nats. Multiplying by T² = 16 keeps the soft gradient at 0.459; without it the gradient would be 0.0287, too small to compete with the hard term.

Set α near 0.7, then drag Temperature T. Minimizing KL to the teacher gives the same gradients as cross-entropy with the soft targets, because the two differ only by the teacher's entropy, which the student cannot change. The gradient on student logit i is (q_i − p_i) / T, and the gap q_i − p_i itself shrinks like 1/T, so the gradient falls like 1/T². Hinton et al. got their best results with a much smaller weight on the hard term. At high T, distillation reduces to matching logits by squared error.

Step 5 of 6: Training a Student

Soft targets vs hard labels on a toy problem

Labelled training points9
Toy: 3 classes, 9 training pointsLines: class boundaries (dashed: teacher)Test loss (900 held-out points)0.40.81.2060120epoch 0hard labelssoft targets
Hard-label test loss
1.099
Soft-target test loss
1.099
Hard-label accuracy
33.3%
Soft-target accuracy
33.3%

Both students start at uniform guesses, test loss 1.099. Press Play or Step to train them on the same 9 points; the teacher's own test loss is 0.458.

Press Play, then repeat with Labelled training points at 6 and at 60. Both students are softmax regression on three Gaussian classes; the teacher is the exact class posterior. With few points the classes are nearly separable, so on hard labels gradient descent keeps growing the weights to push each training probability toward 1. Soft targets have a finite optimum the student can reach. On MNIST, Hinton et al. cut a small net's test errors from 146 to 74 by adding soft targets at T = 20.

Step 6 of 6: Distilling LLMs

Logits, top-k, or the teacher's text

Teacher probabilities kept per token (k)64
Training tokens10B
"The capital of France is" → teacher's next token (illustrative)Paris96%a1.6%the1.0%located0.5%known0.3%not0.2%home0.1%one<0.1%Lyon<0.1%situated<0.1%Kept: 10 of the 10 shown, 100% of their massSequence-levelteacher's tokens onlyTop-k logitsk probabilities per tokenFull logitsall 32,000 per token
Bytes per token
256
Targets for 10B tokens
2.6 TB
vs plain token ids
128×

Keeping the top 64 probabilities costs 256 bytes per token, 2.6 TB for 10B tokens. The tail beyond k is dropped and the kept probabilities are renormalized.

Drag k from 1 to 32,000 and Training tokens up to 1T to see how fast stored targets grow. Kim and Rush (2016) introduced sequence-level distillation for translation, training the student on the teacher's beam-search outputs; their best student ran about 10 times faster with little loss in quality. DistilBERT used soft targets during pretraining and kept about 97% of BERT's language understanding while being 40% smaller. DeepSeek released six dense models, from 1.5B to 70B, made by fine-tuning Qwen and Llama models on outputs from DeepSeek-R1.

What is knowledge distillation?

Knowledge distillation trains a small model, the student, to copy the outputs of a larger trained model, the teacher. Instead of learning only from hard labels, the student learns from the teacher's full probability distribution over classes or tokens. Those probabilities show which wrong answers are close to right, and the student picks up that structure with far fewer parameters.

Distillation is how many compact models are built for serving. DistilBERT is a distilled BERT, Gemma 2's 2B and 9B models were trained on a larger teacher's distributions, and many small open chat and reasoning models are fine-tuned on text written by a bigger model. A smaller student reads fewer bytes per generated token, so it answers faster and costs less to run, and it fits on phones and laptops where the teacher cannot.

Key ideas

  • A student model learns to match a larger teacher's output probabilities instead of only the hard labels.
  • The teacher's small probabilities on wrong classes encode which classes are similar, information a one-hot label throws away.
  • Dividing logits by a temperature T above 1 softens both distributions so those small probabilities affect training.
  • The loss mixes α·T²·KL to the teacher with (1 − α) cross-entropy on the true label; T² keeps the soft gradients from shrinking as T grows.
  • Soft targets act as a regularizer, so a student trained on them overfits less when labelled data is scarce.
  • For LLMs, distillation ranges from matching the teacher's per-token logits to fine-tuning on text the teacher generated.

Common questions

How is knowledge distillation different from quantization and pruning?

Quantization stores the same weights in fewer bits, and pruning removes weights or whole heads from the trained model. Both keep the original architecture. Distillation trains a separate, smaller architecture from scratch or from a smaller checkpoint, using the teacher as the source of targets. The three are often combined, for example a distilled student that is then quantized to 4 bits.

Can a student model be better than its teacher?

Sometimes. When the student has the same size as the teacher, as in born-again networks, it often scores slightly higher, because the soft targets act as a strong regularizer. A much smaller student usually stays below the teacher but well above the same small model trained on labels alone.

Does the student need the same architecture or tokenizer as the teacher?

The architecture can differ freely, since only the outputs are matched. Logit distillation does need the same output space: for LLMs that means a shared vocabulary, so each of the teacher's token probabilities has a matching student token. Sequence-level distillation only needs text, so it works across tokenizers and model families.

Is it allowed to train a model on another model's outputs?

It depends on the teacher's license or terms of service. Several commercial API providers forbid using their outputs to build competing models, while many open-weight licenses permit distillation, sometimes with naming or attribution conditions. Check the specific license before using a model's outputs as training data.

Related explainers

References

  1. Hinton, Vinyals & Dean (2015), "Distilling the Knowledge in a Neural Network"
  2. Kim & Rush (2016), "Sequence-Level Knowledge Distillation"
  3. Sanh et al. (2019), "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter"
  4. Gemma Team (2024), "Gemma 2: Improving Open Language Models at a Practical Size"
  5. DeepSeek-AI (2025), "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning"

Last updated