Adam vs SGD: two ways down the same hill

An optimizer updates a model's parameters, the numbers it learns, to lower the loss, a score for how wrong the model is. SGD (stochastic gradient descent) and Adam both step against the gradient, the slope of the loss, but size each step differently.

1. 1
   Each oval joins points of equal loss, 0.1·p₁² + 10·p₂², with p₁ and p₂ on the param 1 and param 2 axes and the minimum at the green circle. The loss curves 100 times as sharply across param 2 as along param 1. Both start at (2.5, 1.5) with the same noise on every gradient, mimicking mini-batches, random samples of training data. A run stops when a step lands in the target, a loss below 0.02, or after 40 steps.
2. 2
   SGD moves each parameter by the learning rate α times its gradient. On the dashed red path, α = 0.08. The gradient along param 2 is 20·p₂, so, ignoring noise, each step multiplies p₂ by 1 − 0.08 × 20 = −0.6, flipping its sign: the zig-zag. Param 1 shrinks only 1.6% per step and, with the noise, ends at 0.84, loss 0.071.
3. 3
   Adam averages each parameter's recent gradients, called momentum, and their squares. Dividing the first by the root of the second moves each parameter at most about α per step, however steep its direction. The orange path sweeps α from 0.05 to 0.30. At 0.05 the steps are too short: loss 0.22, worse than SGD. From about 0.15 up, momentum, which keeps 90% of its old value each step, carries Adam past the minimum every time. A run only finishes if one step lands inside the target on the way through. So 0.18 to 0.20 hits and misses by turns, and 0.30 ends at param 1 = −1.64, loss 0.40.

More bites

In 1 picture

Why your loss bounces when the learning rate is too high

2 min read