Backprop, activations, optimizers, and learning-rate scheduling
Optimizing by following the slope downhill
Why networks need non-linearity to learn
How networks learn from their mistakes
SGD, momentum, and Adam — smarter gradient descent
Why normalizing activations speeds up deep network training
Step decay, cosine annealing, warmup, and cyclical LR