How Optimizers Work

Vanilla SGD updates each parameter by subtracting α · gradient. Notice the zigzag path — in the steep (param 2) direction it oscillates, while in the flat (param 1) direction progress is slow. Try increasing α to see the oscillation get worse.

Momentum adds a velocity term: v = β·v + ∇L, then θ = θ − α·v. Like a ball rolling downhill, it builds speed in consistent directions and dampens oscillations. Compare the dashed SGD path (>40 steps) to the smoother momentum path (29 steps).

RMSProp tracks the running average of squared gradients per parameter, then divides by √(avg). Parameters with large gradients (steep axis) get smaller effective steps, while small-gradient parameters (flat axis) get boosted. Compare SGD (>40 steps) to RMSProp (30 steps).

Adam combines momentum (first moment) with RMSProp (second moment), plus bias correction for early steps. Compare SGD (>40 steps) to Adam (25 steps) — Adam is the default choice for most deep learning tasks.

Constant LR oscillates near the minimum — steps are too big at the end. Step decay drops the rate at fixed intervals. Cosine annealing smoothly reduces the rate, often giving the best final loss. Most practitioners pair Adam + cosine schedule.

In practice, start with Adam (lr=1e-3) — it works well out of the box. If you need the best generalization (e.g. image classification), try SGD + Momentum with a cosine LR schedule. Use RMSProp for RNNs or reinforcement learning.