Gradient Golf
Each shot is one step of gradient descent on a real loss surface. Get the ball to the lowest point.
Drag on the course and let go to shoot. Direction is your aim, and drag length sets the learning rate. You can see the flag here, which real models never can.
The yellow arrow points straight downhill. Aim close to it and your shot snaps to the slope and turns green.
- Yellow arrow: the steepest way downhill, where the slope is at its most negative. This is the opposite of the gradient, and it may not point at the flag.
- Ring: the slope in every direction from the ball, green for downhill and red for uphill. The white dot is your aim.
- White line: your aim. The dot at its end is where the ball will land.
- Green line: your aim matches the yellow arrow, so they run along the same path.
How to play
The course is a loss surface, and the hole is its lowest point. Drag on the course and let go to shoot. The direction of your drag is your aim, and its length sets the learning rate. How far the ball goes is the learning rate times the slope under it, so the same setting goes further on steep ground. Fewer shots is better, as in real golf.
What each hole teaches
- Open course: the learning rate sets your step. Too small takes forever, and too big overshoots.
- Fog: gradient descent only knows the ground right under it.
- False bottom: following the slope can end in a local minimum.
- Momentum: keeping some speed carries the ball through shallow pits.
- Saddle: a flat point isn't always a minimum, and noise helps the ball escape. Plateaus, which are wide flat areas, slow training down in a similar way.
For the full explanation, read Gradient Descent.
Common questions
Is the ball's movement real gradient descent?
Each shot is computed from a real loss surface. With your aim on the green line and no momentum or noise, the landing spot is exactly one gradient descent step: position minus learning rate times gradient. Aiming elsewhere is a game choice, because real gradient descent has no choice of direction.
What is a local minimum?
A point lower than everything around it, but not the lowest point overall. The slope there is almost zero, so gradient descent stops moving even though better solutions exist elsewhere.
Why doesn't a bigger learning rate get the ball out of a local minimum?
The step is learning rate times slope. In a local minimum the slope is close to zero, so even a large learning rate gives a tiny step. Momentum helps because it carries speed from earlier steps.
What is a saddle point?
A flat point that goes uphill in some directions and downhill in others. The slope is near zero, so plain gradient descent slows to a crawl, while the noise in stochastic gradient descent helps it move away.