Why your loss bounces when the learning rate is too high
The learning rate sets how big a step a model takes each time it updates its parameters. Set it too high and the training loss, the number that measures how wrong the model is, stops falling and bounces. A real network can even hit NaN (not a number, what a computer returns once values overflow and the arithmetic breaks down).
2 min read · Lensa ML
- The curve is the loss for each value of one parameter, θ. Gradient descent starts at the yellow dot, θ = 2.2, and takes 12 steps, each moving θ against the slope, called the gradient, by the gradient times the learning rate α. Near α = 0.1 the blue steps walk down the right wall and stop short of the floor. From about 0.2 to 0.45, they reach the bottom, a loss of −0.145.
- Past about 0.5 the overshoots stop shrinking. The valley's floor is narrow: its curvature h, how sharply it bends, is about 4.1, and gradient descent settles there only when α is below 2/h, about 0.49. Above that, θ rocks back and forth across the floor at θ = −0.29, within about 0.4 of it up to α = 0.6, and never comes to rest.
- Push α toward 0.95 and the first step flings θ far up the left wall. The path bounces but never flies off. Away from the floor, the curve θ² + 0.3 sin 3θ bends with an average curvature of about 2, so the limit there is 2/2 = 1, the slider's maximum. The narrow floor throws θ out, and the walls pull it back. Only in a narrow band near α = 0.92 does it stick, bouncing wall to wall and ending near its starting loss of 4.93. On a plain bowl with α above 2/h, each swing grows until the numbers overflow.
Takeaway
If the loss bounces or turns into NaN early in training, lower the learning rate first. To find a good value, raise the learning rate steadily during a short run while recording the loss, then pick a value a little below where the loss starts to climb.
Go deeper · interactive
Gradient Descent
Drag the learning rate slider and watch the steps start to overshoot the valley.
Open the explainer
