How Gradient Descent Works

You're standing on a loss landscape — the curve shows how wrong the model is for each parameter value θ. The goal: find the valley (lowest loss), but you can't see the whole landscape.

The gradient (derivative) at your position tells you the slope. Move in the negative gradient direction — that's downhill. Positive slope? Go left. Negative slope? Go right.

The learning rate α controls step size. Too small? Painfully slow. Too large? Overshoot and diverge. Just right? Smooth convergence to the minimum.

Each step: compute the gradient, then update θ = θ − α·∇L. Watch the loss decrease as steps grow smaller near the minimum — the gradient shrinks!

With two parameters, the loss surface becomes a landscape. Contour lines show equal-loss regions. Notice the zigzag path — in the steep (θ₂) direction it oscillates, while in the flat (θ₁) direction progress is slow. Real networks do this in millions of dimensions!

Batch GD uses all data — smooth but slow. SGD uses one sample — fast but noisy. Mini-batch is the sweet spot: batches of 32-256 samples give good speed with manageable noise. The noise can even help escape local minima!