Where it settles (mean excess loss, last 50 steps)
0.0444
Best any constant rate gets
0.0220 @ 0.05
Divergence limit 2 / λ_max
0.366
SGD fits a line y = w·x + b to 256 points, but each step only looks at 4 of them, so every gradient is noisy. The orange path is that run from the hollow start towards the green best fit; the bottom chart is how far its loss is above the best possible, on a log scale. At 0.100 it's a compromise: fast enough to arrive and small enough to settle at 0.0444. No constant rate does better than 0.0220: speed early and precision late pull in opposite directions.
Drag the learning rate and compare where the orange run settles. Each step uses a minibatch of 4 points, so the gradient is the true gradient plus noise, and each step moves the weights by rate × noise as well as rate × slope. Near the minimum the slope vanishes but the noise doesn't, so the run keeps bouncing in a region whose size grows with the rate. Small rates shrink that region but take many more steps to arrive, and above 2 / λ_max, set by the steepest direction of the bowl, gradient descent diverges outright.
Step 2 of 6: Big Steps First, Small Steps Later
Step, exponential and cosine decay on the same run
300
Peak rate0.20
Where it settles
0.0044
Constant rate at the same peak
0.123
Steps until it stays below 0.01
234
A schedule changes the learning rate as training goes on (top chart; dashed = constant). Cosine starts at 0.20, so the early steps cover ground as fast as the constant run, then shrinks the rate so the noise ball shrinks with it. It settles at 0.0044 against 0.123 for the constant rate, 28× lower, and below the 0.0220 that the best constant rate can reach. The three shapes differ in detail (step decay drops by 5× twice, exponential shrinks smoothly, cosine stays high longer then falls), but they share the idea: big steps while far away, small steps once close.
Switch between the four schedules at the same peak and compare where each settles. Step decay is the classic ImageNet recipe for ResNets: divide the rate by 10 at fixed epochs (here by 5, twice). Exponential decay shrinks it by a constant factor every step. Cosine annealing (Loshchilov and Hutter, 2016) follows half a cosine from the peak to zero, which keeps the rate high for longer before it falls. In practice the shape matters less than having some decay at all, and cosine with warmup is the common default for transformers.
Step 3 of 6: Decay Too Fast, Decay Too Slow
The schedule has to fit the run's length
300
Decay speed k5.0
Half-life of the rate
42 steps
Rate at the last step
0.0013
Where it settles
0.0019
Final slope w (best 1.45)
1.40
Exponential decay from 0.2, lr = 0.2 · e^(−k·t/300). With k = 5.0 the rate stays high long enough to reach the bottom of the bowl, then falls far enough to settle at 0.0019. The right speed depends on how long the run is, which is why schedules are written in terms of the total number of steps.
Drag the decay speed and watch the half-life and the final slope. If the rate falls too fast, the run freezes before it has crossed the long, shallow direction of the loss bowl, a direction where the curvature is only λ_min ≈ 0.23 and progress per step is slow anyway. If it falls too slowly, the run is still bouncing at the end. Because the right speed depends on the total number of steps, schedules such as cosine are defined over the whole run length, and changing the training budget means changing the schedule.
Step 4 of 6: Big Steps Escape Narrow Valleys
Explore early, settle late, and warm restarts
400
run:
Runs (of 20) ending in the wide valley
0 / 20
Mean final loss (best −1.12)
-0.752
Spread around the wide minimum
0.00
A made-up 1-D loss with a narrow, shallow valley where every run starts, and a wide, deeper one to the right; the gradient is noisy, as in SGD. A small constant rate settles neatly into the narrow valley and stays: 0 of 20 runs escape. The noise kicks are too small to climb out.
Compare the four options and the 20-run counts. A large rate turns gradient noise into big random kicks that can carry the weights out of a narrow valley, and narrow, sharp minima have been linked to worse generalisation (Keskar et al., 2017). Decaying afterwards lets the run settle precisely in whatever wide valley it found. SGDR (Loshchilov and Hutter, 2016) repeats the cosine cycle with warm restarts; here a single high-peak cosine already captures most of the benefit, which matches common practice of a single cosine cycle for most runs.
Step 5 of 6: Warmup Survives a Sharp Start
Why the first steps should be small
Above the red line (2) a step overshoots the bottom by more than it started away.
200
Warmup steps0
Peak rate0.30
Outcome
diverged at step 2
Curvature at the start
28
Peak rate × curvature so far
31.99
Final loss
∞
Another made-up loss, ½w² + ¼w⁴, whose curvature is 28 at the start (w = 3) but only 1 near the minimum. A step is stable only while rate × curvature stays below 2. Starting at the full rate, 0.30 × 28 is far above 2: the first steps overshoot to the other side, where it is even steeper, and the run diverges at step 2. Add warmup steps. Real networks have the same problem: curvature and Adam's noisy early statistics make the first steps the most fragile, so large models warm up for hundreds or thousands of steps.
Drag the warmup steps from 0 upwards at a peak of 0.3, then raise the peak. Gradient descent is stable only while rate × curvature stays below 2, and this loss is 28 times sharper at the start than near the minimum. In real networks the early instability comes from two sources: the loss surface is often sharpest at initialisation, and Adam's second-moment estimates rest on only a few gradients, so its first updates can be far too large. A linear warmup over the first few hundred or thousand steps, as in the original transformer paper, avoids both.
Step 6 of 6: Finding the Peak Rate
The learning-rate range test
Your peak rate0.020
Rate at the lowest loss in the test
0.274
Rate where the loss blows up
0.659
Theory: 2 / λ_max of the problem
0.366
A cosine run from your peak settles at
0.346
Best peak for cosine here
0.12 → 0.0016
Every schedule still needs a peak rate. The range test (Smith, 2017) finds one in a single short run: start tiny, multiply the rate a little every step, and record the loss. Here it falls slowly at first, fastest around 0.01 to 0.1, bottoms out near 0.27, and blows up past 0.66, close to the theoretical limit 2 / λ_max = 0.366 for this problem. Your pick, 0.0200, is far below where the loss falls fastest; a cosine run from it only reaches 0.346 in 300 steps. The test costs a fraction of one training run and needs no theory, which is why it's built into libraries such as fastai and PyTorch Lightning.
Move your peak rate along the range-test curve and compare the cosine run it produces. The range test (Smith, 2017) trains briefly while raising the rate exponentially, and the loss curve reveals the useful range: too small and nothing happens, too large and the loss explodes. A common rule is to take a rate a few times below the point where the loss is lowest. Smith's cyclical learning rates and the one-cycle policy built on the same test, cycling the rate between bounds read off this curve.
What is learning rate scheduling?
Learning rate scheduling is the practice of changing the learning rate during training instead of holding it fixed. A typical schedule starts high so the model makes fast progress, then lowers the rate so it can settle into a minimum without bouncing around it. Many schedules also begin with a short warmup, raising the rate gradually from near zero.
Almost every large model is trained with a schedule. ResNets on ImageNet were classically trained with step decay, dividing the rate by 10 at fixed epochs. Transformers and large language models usually combine linear warmup with cosine or inverse-square-root decay. Schedules are available in every framework, for example torch.optim.lr_scheduler in PyTorch.
Key ideas
A schedule changes the learning rate over the course of training.
High rates make fast early progress; low rates let the model settle into a minimum.
Step decay drops the rate by a fixed factor at set intervals.
Cosine annealing lowers the rate smoothly along a half cosine wave.
Warmup raises the rate gradually at the start to avoid early instability.
Cyclical schedules raise and lower the rate repeatedly instead of only decreasing it.
Common questions
Which learning rate schedule should I use?
For transformers, linear warmup followed by cosine decay to about 10% of the peak is a strong default. For convolutional networks trained with SGD, cosine annealing or step decay both work well. If training is short or the right number of epochs is unknown, reduce-on-plateau adapts to the validation loss.
Do I need a learning rate schedule with Adam?
Usually yes. Adam adapts each weight's step size relative to the others, but the overall rate α still controls how far the model moves. Large models trained with Adam or AdamW almost always use warmup and decay, and skipping warmup is a common cause of early loss spikes.
Should the schedule update per step or per epoch?
Either can work, but per-step updates give a smoother curve and are standard for large models, where one epoch may be the entire training run. Warmup is usually measured in steps, such as the first 1% to 5% of training.
How is the learning rate schedule related to batch size?
Larger batches give less noisy gradients and usually allow a higher peak rate. A common rule of thumb for SGD scales the rate linearly with batch size, which in turn makes warmup more important at the start of training.
Related explainers
Optimizers— SGD, momentum, and Adam — smarter gradient descent