ENID
Lensa ML

How Transfer Learning Works

Step 1 of 6: Training From Scratch Is Expensive

Millions of images, days of compute

Empty model1.2MimagesGPU7 daysof trainingTrained0% — Day 0 of 7Transfer learning skips this

Training a deep network from random weights takes millions of labeled images, days of GPU compute, and careful tuning. Transfer learning skips that by starting from a model someone already trained.

Watch the progress bar crawl through seven simulated days of training. These numbers match the era that made transfer learning standard: AlexNet (2012) trained for five to six days on two GPUs using 1.2 million ImageNet images. Most real projects have a few hundred to a few thousand labeled examples, far too few to train a deep network from random weights without heavy overfitting. Transfer learning carries over what the network learned on a large source task, so the target task only has to learn what is different.

Step 2 of 6: Pretrained Models

Standing on the shoulders of giants

ImageNet-pretrained modelGeneral featuresTask-specific featuresedgestexturesshapespartsobjectsclassesThese features generalize across many visual tasks

Models trained on ImageNet (1.2M images, 1000 classes) learn a hierarchy of features, from edges and textures in the first layers to whole objects near the output.

Hover over each layer, from edges to classes, to read what it detects. Zeiler and Fergus (2013) first showed this hierarchy in a trained CNN by projecting each layer's strongest activations back into pixel space. The final layer is the one part always replaced for a new task, because its 1,000 outputs are tied to ImageNet's classes. Common pretrained backbones include ResNet-50, EfficientNet, and Vision Transformers, and libraries such as torchvision and timm ship them with weights ready to download.

Step 3 of 6: Feature Reuse

Early layers learn universal patterns

Edge DetectorsShallowDiagonalHorizontalVerticalDepth: 1/3Texture DetectorsMidGridDotsStripesDepth: 2/3Part DetectorsDeepCircleRectangleArchDepth: 3/3

Layer 1 detects simple edges. Layer 2 combines edges into textures. Layer 3 assembles textures into parts. Early layers are close to universal; deeper ones grow more specific to the original classes.

Click Layer 1, 2, and 3, or let the display cycle, and compare the detectors at each depth. Yosinski et al. (2014) measured transferability by copying the first n layers of one network into a network trained on a different half of ImageNet's classes. The first layers transferred almost perfectly, while deeper layers transferred worse because they had specialized to the source classes. Frozen copies of middle layers also lost accuracy, but fine-tuning recovered it. Fine-tuned transferred networks even generalized slightly better than networks trained from scratch.

Step 4 of 6: Freeze and Fine-Tune

Keep the base, retrain the top

Freeze layers:
4
Conv1fixedConv2fixedConv3fixedConv4fixedFC1learningOutputlearningFrozen: weights unchangedTrainable: adapts to your dataStandard transfer: freeze base, train top
Frozen layers
4 / 6
Trainable layers
2 / 6
Layers updated
33%

Frozen layers keep their pretrained weights and get no gradient updates. Trainable layers adapt to the new task. The slider sets the boundary: freeze more for small datasets, fewer for large ones.

Drag Freeze layers from 0 to 5 and watch the lock icons move up the network. Freezing a layer means setting requires_grad = False in PyTorch or trainable = False in Keras, so the optimizer skips it. Frozen layers also need no stored gradients, which saves memory. A common recipe trains only the new head first, then unfreezes more layers with a learning rate often ten times smaller. Batch normalization needs care: keeping its running statistics fixed during fine-tuning usually works better on small datasets.

Step 5 of 6: Your Small Dataset Is Enough

Fine-tuning needs far less data

Data required (example)1,000,000From scratch1,000Fine-tuned1000× less dataConvergence (illustrative)EpochsAccuracy0%50%95%92%From scratchFine-tunedEpoch: 0/50

Fine-tuning starts from already-good features, so it converges much faster. In this illustrative sketch, with 1,000 images it passes 90% by epoch 12, while the scratch model trained on 1,000,000 images is still climbing. Each of its epochs is also 1000× more work. The curves show the typical shape, not a measured run.

Watch the two curves race for 50 epochs. The fine-tuned model starts near 60% before any training on the new task, because its features already separate many classes. The scratch model climbs slowly and reaches only about 80% by epoch 50. Data needs vary by task, but a few hundred labeled images per class is often enough when the new images resemble the pretraining set. Data augmentation, such as random flips and crops, stretches a small dataset further and pairs well with fine-tuning.

Step 6 of 6: When to Transfer

Practical guide to transfer learning decisions

Dataset size →Domain similarity →SmallLargeHighLowFine-Tune TopSmall data, similar domainFine-Tune AllLarge data, similar domainFreeze AllSmall data, different domainTrain From ScratchLarge data, different domain
Hover a quadrant to see the recommended strategy

The right strategy depends on two factors: dataset size, and how close the task is to the pretraining domain. Small + similar? Fine-tune the top. Large + different? Pretrained weights matter least.

Hover each quadrant of the dataset size and domain similarity chart to read its strategy. Domain similarity covers the inputs as well as the labels. Medical scans and satellite images differ from ImageNet photos in color, texture, and scale, so more layers need retraining. Raghu et al. (2019) found that ImageNet pretraining gave little accuracy gain on medical imaging tasks, although it still sped up convergence. For text, the same chart applies with a pretrained language model such as BERT in place of the CNN.

What is transfer learning?

Transfer learning is a technique where a model trained on one task is reused as the starting point for a different but related task. Instead of starting from random weights, the new model starts from weights that already encode useful features, such as edges and shapes for images or grammar and word meaning for text. Only part of the network, or all of it with a small learning rate, is then trained on the new data.

Transfer learning is the default way to build vision and language models when labeled data is limited. A classifier for skin lesions, crop diseases, or factory defects usually starts from a network pretrained on ImageNet. Text classifiers, chatbots, and code assistants start from pretrained language models such as BERT or GPT. It cuts the data needed from millions of examples to thousands or fewer, and training time from days to hours.

Key ideas

  • Transfer learning starts a new task from a model already trained on a large related dataset.
  • Early CNN layers learn general features such as edges and textures, while later layers become task-specific.
  • Freezing keeps a layer's weights fixed, and fine-tuning updates them with a small learning rate.
  • Smaller datasets call for freezing more layers, and larger datasets allow fine-tuning more of the network.
  • Fine-tuning reaches good accuracy with far less labeled data and compute than training from scratch.
  • Pretrained language models such as BERT and GPT apply the same idea to text.

Short takes on this topic

Common questions

What is the difference between transfer learning and fine-tuning?

Transfer learning is the general idea of reusing a model trained on one task for another. Fine-tuning is one way to do it: continue training some or all of the pretrained weights on the new data. The other common way is feature extraction, where the pretrained network stays frozen and only a new classifier on top is trained.

Does transfer learning work for NLP?

Yes, and it is now the standard approach. ULMFiT (2018) showed that fine-tuning a pretrained language model beat task-specific models on text classification, and BERT and GPT made the pattern universal. Almost every modern NLP system starts from a pretrained Transformer.

What is negative transfer?

Negative transfer is when starting from pretrained weights gives worse results than training from scratch. It can happen when the source and target domains are very different, or when an aggressive learning rate destroys the pretrained features early in fine-tuning. Comparing against a scratch baseline on a validation set is the way to detect it.

What is domain adaptation?

Domain adaptation is a form of transfer learning where the task stays the same but the input distribution changes, for example a sentiment model trained on movie reviews applied to product reviews. Methods range from fine-tuning on a small labeled sample of the new domain to training on unlabeled target data so that features from both domains look alike.

Related explainers

References

  1. Yosinski et al. (2014), "How transferable are features in deep neural networks?"
  2. Zeiler & Fergus (2013), "Visualizing and Understanding Convolutional Networks"
  3. Howard & Ruder (2018), "Universal Language Model Fine-tuning for Text Classification" (ULMFiT)
  4. Raghu et al. (2019), "Transfusion: Understanding Transfer Learning for Medical Imaging"
  5. Stanford CS231n, "Transfer Learning" course notes

Last updated