Transfer learning: freeze 99.9% of a model, train a new last layer

Transfer learning reuses a model trained on one task as the starting point for another. ResNet-50 learned from 1.2 million labeled ImageNet photos. A new image task often has a few hundred photos per class, too few to train such a network from scratch, but often enough to train one new layer on top of it.

1. 1
   ResNet-50 is a 50-layer convolutional network, a standard image design that slides small learned filters across a photo, trained to sort ImageNet photos into 1,000 classes. Here a weight means any learned number, biases included. Its body, a stem and four stages of convolution blocks, turns a photo into 2,048 numbers that describe it: edges and textures in early layers, object parts in later ones. The head, its last layer, maps those 2,048 numbers to 1,000 class scores with 2,049,000 weights.
2. 2
   For a task with 10 classes, such as 10 crop diseases, the 1,000-class head is removed. A new 2,048 to 10 layer takes its place: 2,048 × 10 weights plus 10 biases, 20,490 in all. The body is frozen, meaning its 23,508,032 weights keep their ImageNet values and training never updates them. Counted from the layer shapes, body and old head add up to 25,557,032, the count torchvision lists.
3. 3
   The new model holds 23,528,522 weights. Training changes 20,490 of them, 0.09%, while 99.91% are reused as they are. On the bar drawn to scale, the head is a sliver at the right end. Gradients, the error signals training sends backward through the network, stop at the head, so frozen layers need none. Each step costs less, though every photo still runs forward through the frozen body.