Curse of Dimensionality
Volume Explosion
Hypercube surprise
In low dimensions, the inscribed ball fills a good fraction of the cube — most of the volume is near the center.
Step 1 of 6: Volume Explosion
Hypercube surprise
In low dimensions, the inscribed ball fills a good fraction of the cube — most of the volume is near the center.
Step 2 of 6: Distance Concentration
Everything is far away
Distances are spread out — nearby and far points are clearly different.
Step 3 of 6: Data Sparsity
You need exponentially more data
With 1000 points, you have good coverage. Each region of the space is well-sampled.
Step 4 of 6: k-NN in High Dimensions
Neighborhoods grow huge
Small radius captures local neighbors — k-NN works well in low dimensions.
Step 5 of 6: Irrelevant Features
Noise drowns out signal
Clean separation with just the relevant features — classifier works perfectly.
Step 6 of 6: Mitigation Strategies
Reducing dimensions
Still too many dimensions — most features add noise without useful information.