Why gradients vanish in deep networks, in 1 picture

A neural network learns by nudging its weights, the numbers that connect its layers, to lower the loss, a measure of how wrong it is. The gradient tells each weight which way to move and how far. In a deep network, that signal can shrink to almost nothing before it reaches the first layers, so they barely learn.

1. 1
   The clip shows a network: 2 inputs, hidden layers of 3 units each, and 1 output. Each unit passes the weighted sum of its inputs through a sigmoid, an S-shaped function with outputs between 0 and 1. Backpropagation, the method that computes gradients, starts at the output with a gradient of 1.000 and passes it back layer by layer. The bars show the average gradient size at each layer, input on the left.
2. 2
   With 1 hidden layer, the gradient reaching the inputs averages 0.0154. Each extra layer cuts it again: 9.1 × 10⁻⁴ with 2 hidden layers, 1.1 × 10⁻⁴ with 3, 5.7 × 10⁻⁵ with 4 and 3.9 × 10⁻⁶, about 4 millionths, with 5. At 5 hidden layers the inputs get a gradient about 260,000 times smaller than the output's, while the last hidden layer still gets 0.088.
3. 3
   Each step back multiplies the gradient by the layer's weights and by the sigmoid's slope at each unit. That slope is at most 0.25, at the middle of the S, and the weights here lie between −0.8 and 0.8. Here each step back keeps only about 6% to 32% of the average gradient. A product of many numbers below 1 shrinks fast, and every added layer adds another factor, so early layers get far smaller updates than late ones.

More bites

In 1 picture

Why your loss bounces when the learning rate is too high

2 min read

Plain words

Batch normalization in plain words

2 min read