Convexity, Minima & Saddle Points
Convex vs Non-Convex
Why shape matters for optimization
A convex function curves upward everywhere, forming a single bowl. Any local minimum is automatically the global minimum -- optimization is straightforward.
Step 1 of 6: Convex vs Non-Convex
Why shape matters for optimization
A convex function curves upward everywhere, forming a single bowl. Any local minimum is automatically the global minimum -- optimization is straightforward.
Step 2 of 6: Local vs Global Minima
Where you start determines where you end
From x=2.0, gradient descent follows the slope into a local minimum at x=1.6. It can't escape — the gradient is zero here. The true global minimum is at x=-0.5. Try moving the start to a different colored basin.
Step 3 of 6: Saddle Points
Flat but not a minimum
Near the saddle point: the gradient is close to zero, but this is NOT a minimum. f(x,y) = x^2 - y^2 curves up in x and down in y. Gradient-based methods can stall here.
Step 4 of 6: The Convexity Test
Second derivatives reveal curvature
f''(x) < 0 here (dashed pink below zero). The curve bends downward -- this region is concave. The function is NOT globally convex.
Step 5 of 6: Loss Landscapes
Neural networks in the wild
A moderate-width network. The landscape has some bumps but is becoming smoother as width increases.
Step 6 of 6: Escaping Local Minima
Momentum and learning rate to the rescue
Moderate momentum accumulates velocity from past gradients, helping push through shallow local minima while staying stable.