Lensa ML
Lensa ML

Curse of Dimensionality

Step 1 of 6

Volume Explosion

Hypercube surprise

Dimensions (d)2
d-cube78.5%d = 2 dimensions
DIMENSIONS
2
BALL / CUBE
78.54%
BALL VOL
3.142

In low dimensions, the inscribed ball fills a good fraction of the cube — most of the volume is near the center.

Step 1 of 6: Volume Explosion

Hypercube surprise

Dimensions (d)2
d-cube78.5%d = 2 dimensions
DIMENSIONS
2
BALL / CUBE
78.54%
BALL VOL
3.142

In low dimensions, the inscribed ball fills a good fraction of the cube — most of the volume is near the center.

Step 2 of 6: Distance Concentration

Everything is far away

Dimensions (d)2
Pairwise DistanceCount50848010586300.00.20.40.60.70.9d = 2
DIMENSIONS
2
MIN DIST
0.02
MAX DIST
1.09
MAX / MIN
59.36

Distances are spread out — nearby and far points are clearly different.

Step 3 of 6: Data Sparsity

You need exponentially more data

Dimensions (d)1
Comparisonlog₁₀(samples)10^110^210^310^410^510^610^710^810^910^1010^1Data Needed1000Available
DIMENSIONS
1
DATA NEEDED
10
AVAILABLE
1,000
DENSITY
100.00

With 1000 points, you have good coverage. Each region of the space is well-sampled.

Step 4 of 6: k-NN in High Dimensions

Neighborhoods grow huge

Dimensions (d)2
feature spacequeryr=0.224d = 2 — neighborhood radius to find 5 of 100 points
DIMENSIONS
2
RADIUS (k=5)
0.224
SPACE COVERED
5.0%

Small radius captures local neighbors — k-NN works well in low dimensions.

Step 5 of 6: Irrelevant Features

Noise drowns out signal

Irrelevant features0
2D View (Relevant Features)Accuracy vs Noise Features50%60%70%80%90%100%0510152098%noise features
RELEVANT
2
IRRELEVANT
0
TOTAL
2
ACCURACY
98%

Clean separation with just the relevant features — classifier works perfectly.

Step 6 of 6: Mitigation Strategies

Reducing dimensions

Target dimensions (after reduction)20
Accuracy vs. Reduced Dimensions50%60%70%80%90%100%2d5d8d11d14d17d20dsweet spot60%target dimensionsPCAFeature SelectionAutoencoders
ORIGINAL D
20
TARGET D
20
ACC BEFORE
60%
ACC AFTER
60%

Still too many dimensions — most features add noise without useful information.