ENID
Lensa ML

Vanishing & Exploding Gradients

Step 1 of 6: Gradient Flow

Forward & backward pass through a real network

Hidden layers3 (2–3–3–3–1)
inputh1h2h3output
avg |∂L/∂a|1e-4in9e-4h10.006h20.063h31.000out
Click a neuron to see its forward & backward calculations

Architecture

2–3–3–3–1

Avg Input Grad

1.1e-4

Output Act

0.413

With 3 hidden layers, the gradient magnitude weakens toward the input. Each layer multiplies it by weight × σ', and σ' ≤ 0.25, so with weights this small (|w| ≤ 0.8) every layer shrinks it.

Drag the hidden layers slider from 1 to 5, switch shapes, and tap a node to see its calculation. In matrix form, the gradient reaching layer 1 is the output gradient multiplied by one Jacobian per layer, and each Jacobian combines that layer's weights with its activation derivatives. The size of this product is roughly the product of the Jacobians' sizes. Sepp Hochreiter analyzed the effect in his 1991 diploma thesis, and Bengio, Simard, and Frasconi (1994) showed that it makes long-range dependencies hard for recurrent networks to learn by gradient descent.

Step 2 of 6: Exploding Gradients

Linear network where gradients blow up backward

Hidden layers3 (2–3–3–3–1)
Weight scale1.5×
inputh1h2h3output
avg |∂L/∂a|0.3in0.4h10.4h20.5h31.0out
Click a neuron to see its forward & backward calculations

Architecture

2–3–3–3–1

Avg Input Grad

0.35

Status

Mild

With weight scale 1.5× and 3 hidden layers, gradients are still manageable. Increase either and they blow up.

Drag the weight scale from 0.5 to 3 and add hidden layers. This network has no activation function, so each layer's Jacobian is just its weight matrix. If a typical layer multiplies the gradient by about 2, then 6 layers multiply it by 2⁶ = 64, and 20 layers by about a million. Recurrent networks are especially exposed because the same weight matrix is applied at every time step, so a sequence of 100 tokens raises it to the 100th power. When its largest singular value exceeds 1, gradients can explode.

Step 3 of 6: Sigmoid vs ReLU

Why activation choice decides gradient fate

Input x0.00
Layers5
-4-2240.251.0
σ'(x), max 0.25ReLU'(x), 0 or 1
Derivative product over 5 layers (same x, weights left out): (deriv)^5σ0.0010ReLU0.0

σ'(x)

0.250

ReLU'(x)

0

σ after 5L

0.0010

ReLU after 5L

0.0

At x=0, sigmoid has its largest derivative (0.25) and ReLU is right at its kink. Even at its peak, sigmoid shrinks gradients 4× per layer.

Drag the input x and the layers slider from 1 to 10, and watch the two "after" cards. The cards multiply the same derivative once per layer, which is the best case for sigmoid. Even then, 10 layers at the peak leave 0.25¹⁰, about one millionth of the gradient. ReLU's derivative is exactly 1 or 0, so a chain of active units passes the gradient unchanged. Leaky ReLU, ELU, and GELU keep a small nonzero slope for negative inputs, so a unit that drifts negative can still recover.

Step 4 of 6: Training Impact

How gradient problems stall real training

Depth5 layers (2–3–3–3–1)
Learning rate1.0
0.50epoch (real gradient descent, 2–3–3–3–1 sigmoid network, 4 training samples)2L5Lloss stalls: early layers barely updateavg |Δw|5e-5h10.0003h20.0030h30.0701out

Final Loss (5L)

0.1327

Final Loss (2L)

0.0239

L1 avg |Δw|

5.2e-5

With 5 sigmoid layers, the loss stalls at 0.133 while a 2-layer net reaches 0.024. The first hidden layer's update is only 5.2e-5, effectively frozen. Vanishing gradients have stalled training.

Drag depth from 2 to 8 and the learning rate from 0.1 to 3, and compare the two final-loss cards. The 4 targets are 0.9, 0.1, 0.8, and 0.2, so a network that outputs 0.5 for every example already scores about 0.125. Deep sigmoid nets here settle near that value, which means they have learned almost nothing. A rate of 3 rescues the 4-layer net, but a higher rate scales every layer's update by the same factor. It cannot close a gap of several hundred times between the first and last layers.

Step 5 of 6: Skip Connections

How identity shortcuts rescue gradient flow

Hidden layers: 5(2–3–3–3–3–3–1)
inputh1h2h3h4h5output
avg |grad|no skipskip4e-60.01in5e-50.08h14e-40.01h20.000.08h30.030.03h40.090.08h51.01.0out
Click a neuron to see its forward & backward calculations

Input Grad (no skip)

3.9e-6

Input Grad (skip)

0.0121

Improvement

3e+3×

Without skip connections, the gradient must pass through every sigmoid (σ' ≤ 0.25). After 5 layers, the input gradient is 3.9e-6. Toggle "Skip Connections" to see how residual blocks rescue gradient flow.

Toggle Skip Connections and compare the input gradient with and without them. A residual block computes y = x + F(x), so its Jacobian is I + ∂F/∂x. The identity term passes the gradient back unchanged even when ∂F/∂x is tiny. He et al. (2015) used this design to train ResNets with 152 layers, far deeper than earlier convolutional networks. Every transformer layer uses the same pattern around its attention and feed-forward sublayers. The LSTM cell state works the same way through time: it is updated by addition, not repeated multiplication.

Step 6 of 6: Gradient Clipping

The safety net for exploding gradients

Max norm: 5.0
Before3.2-7.15.5-2.88.3-6.0After1.1-2.51.9-1.02.9-2.1

Original ‖g‖

14.27

Threshold

5.0

Clipped ‖g‖

5.00

Gradient norm 14.3 exceeds threshold 5.0. All components are scaled by 0.350 to bring the norm down, preserving direction but limiting magnitude.

Drag the max norm slider from 0.5 to 10 and compare the bars before and after. The gradient vector here has norm √(3.2² + 7.1² + 5.5² + 2.8² + 8.3² + 6.0²) ≈ 14.3, so a threshold of 5 scales every component by about 0.35. That is clipping by norm, from Pascanu et al. (2012). Clipping by value caps each component separately, which can change the gradient's direction. Large language model training commonly clips the global norm across all parameters at 1.0.

What are vanishing and exploding gradients?

Vanishing and exploding gradients are two failures of training deep networks, where the gradient shrinks toward zero or grows without bound as backpropagation carries it from the output back toward the input. With vanishing gradients, the early layers barely change and the network learns slowly or not at all. With exploding gradients, updates become huge, the loss jumps around, and the weights can overflow to NaN.

Both problems come from the same cause: the gradient at an early layer is a product of many per-layer factors, and a long product of numbers below 1 shrinks while one above 1 grows. They held back deep feed-forward networks and recurrent networks for years. The modern fixes are ReLU-family activations, careful weight initialization, normalization layers, residual connections, gated RNN cells such as LSTMs, and gradient clipping.

Key ideas

  • An early layer's gradient is a product of one factor per later layer.
  • Factors consistently below 1 make gradients vanish; factors above 1 make them explode.
  • Vanishing gradients leave early layers almost untrained while later layers keep learning.
  • Exploding gradients cause erratic loss, huge updates, and NaN values.
  • Residual connections add an identity path that carries the gradient past each block.
  • Gradient clipping rescales an oversized gradient while keeping its direction.

Short takes on this topic

Common questions

How do I tell if my network has vanishing or exploding gradients?

Log the gradient norm of each layer during training. Vanishing gradients show up as norms in early layers that are orders of magnitude smaller than in later layers, with a loss that plateaus early. Exploding gradients show up as sudden spikes in the loss or gradient norm, or as NaN values.

Do vanishing gradients still happen in modern networks?

Far less often. ReLU-family activations, He or Xavier initialization, normalization layers, and residual connections together keep gradients in a workable range even in networks with hundreds of layers. Plain recurrent networks on long sequences remain vulnerable, which is one reason transformers replaced them.

Why do RNNs suffer more than feed-forward networks?

Backpropagation through time multiplies by the same recurrent weight matrix once per time step. Over 100 steps, a matrix whose largest singular value is slightly below 1 shrinks the gradient toward zero, and one slightly above 1 can make it explode. LSTMs and GRUs add gated, additive paths to avoid this.

What value should I use for gradient clipping?

A max norm of 1.0 is a common default for transformers and RNNs. A good value sits slightly above the gradient norms seen in normal training, so clipping only triggers on outliers. Logging gradient norms for a few hundred steps shows where that is.

Related explainers

References

  1. Bengio, Simard & Frasconi (1994), "Learning long-term dependencies with gradient descent is difficult", IEEE Transactions on Neural Networks
  2. Glorot & Bengio (2010), "Understanding the difficulty of training deep feedforward neural networks", AISTATS
  3. Pascanu, Mikolov & Bengio (2012), "On the difficulty of training Recurrent Neural Networks"
  4. He et al. (2015), "Deep Residual Learning for Image Recognition"
  5. Zhang et al., "Dive into Deep Learning", section 5.4: Numerical Stability and Initialization

Last updated