How Learning Rate Scheduling Works

High learning rate: loss oscillates as the model overshoots the minimum repeatedly.

Moderate step decay gives a nice staircase pattern. Each plateau lets the model settle before reducing the step size.

Moderate decay rate produces a smooth curve that gradually reduces exploration over training.

Moderate cycle length creates a smooth annealing curve. The cosine shape spends more time at low LRs for fine-tuning.

Moderate warmup lets the model build reliable gradient statistics before applying the full learning rate. Standard for transformers.

Moderate cycles balance exploration and convergence. The triangular wave provides natural warmup and cooldown within each cycle.