Overfit Tailor
Fit a curve to noisy dots, then see how it does on points it has never seen. A perfect fit is usually a trap.
- Training points
- Your curve
Drag the handles to shape the curve through the dots, then test it on points you can't see. Your score comes from those hidden points, not the dots in front of you.
Training error only measures the dots you can see. The next round shows why you can't trust it.
How to play
Each round shows noisy training points. Shape a curve, then test it on hidden points that come from the same pattern. Your test error has to beat the target, and you have three tries per round. All the points are fixed, so every player sees the same data.
What each round teaches
- Hand fit: a model is scored on points it hasn't seen, not on the dots it was fitted to. Nine handles can't bend enough to memorize much noise, so the next round gives the curve more freedom.
- Degree: training error always falls as a model gets more flexible, while test error falls and then rises.
- Few points: with little data, regularization keeps a flexible model under control.
For the full explanation, read Overfitting.
Common questions
What is overfitting?
A model overfits when it learns the noise in its training data along with the real pattern. Its training error looks excellent, but it does worse on data it hasn't seen.
Why can't training error tell you when to stop?
A more flexible model can always match its training points at least as well, so training error keeps falling. Only data the model hasn't seen shows when added flexibility has stopped helping.
What is underfitting?
A model underfits when it's too simple to follow the pattern. Both training error and test error stay high, as with a straight line through curved data.
What does regularization do?
It adds a penalty for large weights to the loss, so the fit prefers calmer curves. This game uses ridge regression, also called L2 regularization. Too little leaves the wild swings, and too much flattens the real pattern.