ENID
Lensa ML

How Optimizers Work

Step 1 of 6: Vanilla SGD

Take a step proportional to the gradient

param 1 →param 2 ↑minstart
Learning rateα = 0.080
Steps
>40
Final loss
0.0710

Vanilla SGD updates each parameter by subtracting α · gradient. The zigzag path comes from the two directions: along the steep param 2 axis it oscillates, while along the flat param 1 axis progress is slow. Increase α and the oscillation gets worse.

Drag the learning rate α between 0.005 and 0.095 and compare the final loss. The loss here is 0.1·x² + 10·y², so the curvature along param 2 is 100 times that along param 1. That ratio is the condition number, and it is what makes the path zigzag. Plain gradient descent is stable only if α < 2 divided by the largest curvature, here 2/20 = 0.1, which is why the slider stops just below it. The noise added to each gradient stands in for mini-batch sampling, the "stochastic" in SGD.

Step 2 of 6: Momentum

Roll downhill like a ball, building speed

param 1 →param 2 ↑minstart
SGDMomentum
Momentum (β)β = 0.75
SGD steps
>40
Momentum steps
29

Momentum adds a velocity term: v = β·v + ∇L, then θ = θ − α·v. Like a ball rolling downhill, it builds speed along consistent directions and damps oscillations. Compare the dashed SGD path (>40 steps) to the smoother momentum path (29 steps).

Drag β from 0 to 0.99 and compare the momentum path with the dashed SGD path. On a slope with a steady gradient, the velocity grows until each step is α/(1 − β) times the gradient. At β = 0.75 and α = 0.04 that is an effective rate of 0.16, four times the base rate. Oscillating components cancel out instead. This heavy-ball method dates to Polyak (1964). Nesterov momentum evaluates the gradient at the look-ahead point, and β = 0.9 is the usual default.

Step 3 of 6: RMSProp

Adapt the learning rate per parameter

param 1 →param 2 ↑minstart
SGDRMSProp
Decay (γ)γ = 0.90
SGD steps
>40
RMSProp steps
30

RMSProp tracks the running average of squared gradients per parameter, then divides by √(avg). Parameters with large gradients (steep axis) get smaller effective steps, while small-gradient parameters (flat axis) get boosted. Compare SGD (>40 steps) to RMSProp (30 steps).

Drag the decay γ and compare the step counts on the cards. RMSProp keeps s = γ·s + (1 − γ)·g² for each parameter and divides the step by √s, so each weight's effective rate is roughly α/√s. A steep axis gets a small rate and a flat axis a large one. Geoffrey Hinton proposed it in a 2012 Coursera lecture rather than a paper. It fixed a flaw in its predecessor AdaGrad, which sums squared gradients forever, so its step sizes only shrink and training eventually stalls.

Step 4 of 6: Adam — Best of Both

Momentum + adaptive rates = the default optimizer

param 1 →param 2 ↑minstart
SGDAdam
Learning rate (α)α = 0.200
SGD steps
>40
Adam steps
25

Adam combines momentum (first moment) with RMSProp (second moment), plus bias correction for early steps. Compare SGD (>40 steps) to Adam (25 steps). Adam is the default choice for most deep learning tasks.

Drag the learning rate α and compare Adam's step count with SGD's. Adam keeps m, an average of gradients with β1 = 0.9, and v, an average of squared gradients with β2 = 0.999, then steps by α·m̂/(√v̂ + ε). Both averages start at zero, so early on they underestimate. After the first step, m = 0.1·g, and bias correction divides by 1 − 0.9¹ = 0.1 to recover g. Because m̂/√v̂ is near ±1 on a steady slope, each weight moves about α per step whatever its gradient's scale.

Step 5 of 6: Learning Rate Schedules

Slow down as you approach the minimum

epochloss025507510000.51
ConstantStep DecayCosine

Constant LR keeps the same step size the whole time. Step decay drops the rate at fixed intervals. Cosine annealing smoothly reduces the rate, so its curve settles and stops wobbling. These curves are an illustrative sketch, not a real training run, and they show the trade-off: decay too early and progress slows, so here constant LR ends lowest. In real training with noisy gradients, a constant rate keeps bouncing around the minimum, which is why decaying schedules usually give the best final loss. A common pairing is Adam with a cosine schedule.

Click each schedule button and compare the loss curves near epoch 100. Step decay here halves the rate every 30 epochs. Cosine annealing sets the rate to α·½(1 + cos(πt/T)), which falls slowly at first, fastest in the middle, and slowly again near the end. A schedule and an optimizer work together: Adam adapts the step per weight, while the schedule scales every step over time. The learning rate scheduling explainer covers warmup, restarts, and cyclical schedules in detail.

Step 6 of 6: Choosing an Optimizer

When to use what — a practical guide

StartBest generalization& time to tune?yesSGD + Mom.noRNN orRL task?noAdamyesRMSPropdefault choice
SGD + Momentum
+ Simple, generalizes well − Needs LR tuning
RMSProp
+ Good for RNNs, non-stationary − No bias correction
Adam
+ Fast convergence, little tuning − May generalize worse

In practice, start with Adam (lr=1e-3), which works well out of the box. If you need the best generalization (e.g. image classification), try SGD + Momentum with a cosine LR schedule. Use RMSProp for RNNs or reinforcement learning.

Follow the flowchart from Start and read the pros and cons below it. Wilson et al. (2017) found that adaptive methods such as Adam sometimes reached solutions that generalized worse than tuned SGD on image tasks, which is why SGD with momentum is still common there. For transformers, Adam-family optimizers are the norm, and AdamW is the usual choice because it decouples weight decay from the adaptive step. Memory matters at scale: for a 7-billion-parameter model, Adam's two state buffers add 14 billion stored values.

What is an optimizer in deep learning?

An optimizer is the rule that turns gradients into weight updates during training. Plain stochastic gradient descent (SGD) moves each weight a fixed fraction of its gradient. Momentum, RMSProp, and Adam add memory of past gradients to move faster along consistent directions and to give each weight its own effective step size.

The optimizer affects how quickly a model trains, which learning rates it tolerates, and sometimes how well it generalizes. Image classifiers such as ResNets are often trained with SGD plus momentum. Transformers and large language models are almost always trained with Adam or its variant AdamW. Optimizer state also costs memory: Adam stores two extra numbers for every parameter.

Key ideas

  • SGD updates each weight by the learning rate times its gradient.
  • Momentum averages past gradients, speeding up consistent directions and damping oscillation.
  • RMSProp divides each step by a running root-mean-square of that weight's gradients.
  • Adam combines momentum with RMSProp-style scaling and corrects the bias of both averages.
  • Ill-conditioned losses, steep in some directions and flat in others, are where adaptive methods help most.
  • Adam and AdamW are the defaults for transformers; SGD with momentum remains common for vision.

Short takes on this topic

Common questions

What is the difference between Adam and AdamW?

Adam implements weight decay by adding it to the gradient, so the adaptive scaling also shrinks the decay for weights with large gradients. AdamW subtracts the decay from the weights directly, separate from the adaptive step. That makes regularization behave as intended, and AdamW is now the standard choice for training transformers.

What learning rate should I use with Adam?

The paper's default of 0.001 is a reasonable start for small models. Large transformers usually use smaller peak rates, from about 1e-4 to 6e-4, with warmup and decay. Fine-tuning a pretrained model often uses 1e-5 to 5e-5.

What does epsilon do in Adam?

Epsilon, 1e-8 by default, is added to the denominator √v̂ to avoid dividing by zero for weights whose gradients are near zero. It also caps the effective step for those weights. Some models raise it, for example to 1e-6, to make training more stable in low-precision arithmetic.

Is stochastic gradient descent the same as mini-batch gradient descent?

In deep learning the terms are used interchangeably. Strictly, SGD estimates the gradient from one example, and mini-batch gradient descent averages it over a small batch such as 32 or 256 examples. Every modern framework's "SGD" optimizer works on whatever batch it is given.

Related explainers

References

  1. Kingma & Ba (2014), "Adam: A Method for Stochastic Optimization"
  2. Loshchilov & Hutter (2017), "Decoupled Weight Decay Regularization"
  3. Wilson et al. (2017), "The Marginal Value of Adaptive Gradient Methods in Machine Learning"
  4. Goodfellow, Bengio & Courville (2016), "Deep Learning", chapter 8: Optimization for Training Deep Models
  5. Zhang et al., "Dive into Deep Learning", section 12.10: Adam

Last updated