1Data Preprocessing 2Feature Extraction 3Dimensionality Reduction 4Embeddings 5Transfer Learning

Lensa ML

How Dimensionality Reduction Works

Step 1 of 6: The Curse of Dimensions

More features means exponentially more space

As dimensions increase, the same 12 points become exponentially sparser. In high dimensions most of the space is empty, which is the curse of dimensionality.

Click 1D, 2D, and 3D, or let the display cycle, and watch the same 12 points spread out. To keep the spacing of 12 points on a line, a square needs 144 points and a cube 1,728, since the count grows as 12^d. Distances also lose contrast. In 100 dimensions, the nearest and farthest neighbors of a random point are almost equally far away, which hurts KNN and clustering. Real data usually escapes the worst of this because it lies near a lower-dimensional surface, and dimensionality reduction tries to find that surface.

Step 2 of 6: PCA — Find the Best Angle

Rotate the data to maximize variance

Rotation angle

56°

Explained variance

83.2%

Variance on PC1

2.96

PCA picks the axis that captures the most spread (variance) in the data. Rotate the line to maximize the explained variance; that angle is the first principal component.

Drag the Rotation angle from 0° to 180° and watch the explained variance readout rise and fall. Projected onto a unit direction w, the points have variance wᵀCw, where C is the covariance matrix, and PC1 is the w that makes this largest. That direction is the top eigenvector of C, and its eigenvalue is the variance along it. PC2 is the best direction perpendicular to PC1, and so on. In this cloud the readout peaks near 34°, where PC1 holds about 96% of the variance. Features in large units dominate unless the data is standardized first.

Step 3 of 6: Choosing Dimensions

How many components to keep?

Components to keep

3 of 6

Variance kept

82%

Recon. error

18%

The scree plot shows the variance each component captures. Look for the elbow, where extra components add little. Fewer components mean more compression and higher reconstruction error.

Drag Components to keep from 1 to 6 and watch the cumulative curve and the Variance Kept readout. Besides the elbow, two rules are common. One keeps enough components for a fixed share of variance, often 90% or 95%. The other, the Kaiser rule for standardized data, keeps components with an eigenvalue above 1. With the scree values shown, 90% needs 4 components and 95% needs 5. When PCA feeds a model, the most reliable choice is to treat the count as a hyperparameter and pick it by cross-validation.

Step 4 of 6: PCA in Action

Compress and reconstruct

PCA compresses 5 features into 2 components, then reconstructs the original from them. The red error shows how much information was lost. Simplified: the three components here are a fixed orthonormal basis standing in for the eigenvectors that real PCA would compute from many centred samples.

Click 1 comp., 2 comp., and 3 comp. and compare the original and reconstructed bars. Compression computes scores z = Wᵀ(x − x̄), and reconstruction maps back with x̂ = Wz + x̄, where the columns of W are the kept components. Because the components are orthonormal, the squared reconstruction error equals what lies along the dropped directions, so it can only fall as components are added: here it goes from 0.041 with one component to 0.004 with three. Denoising uses the same round trip: noise spread thinly across many directions mostly lands in the dropped ones. An autoencoder is the non-linear version of this idea.

Step 5 of 6: t-SNE — Preserving Neighborhoods

Keeping similar points close in 2D

Perplexity

t-SNE maps high-dimensional data to 2D while keeping local neighborhoods. Similar points stay close. Perplexity sets roughly how many neighbors each point considers; higher values bring out broader structure. The right panel is a schematic, not a real t-SNE run: here the slider only tightens the clusters.

Drag Perplexity from 5 to 30 and watch the clusters tighten in this schematic. t-SNE turns distances into probabilities. In the original space, each point's neighbors get weights from a Gaussian whose width is tuned so the effective number of neighbors matches the perplexity. In 2D it uses a heavy-tailed Student-t distribution instead, which lets moderately distant points sit far apart and keeps clusters from crushing together. It then moves the 2D points to minimize the KL divergence between the two distributions. The paper suggests perplexities from 5 to 50.

Step 6 of 6: PCA vs t-SNE

Linear simplicity vs non-linear beauty

	PCA	t-SNE
Type	Linear	Non-linear
Speed	Fast	Slow
Distances	Largely kept	Distorted
Best for	Preprocessing	Visualization

Use PCA for fast, invertible reduction (preprocessing, denoising). Use t-SNE to visualize clusters, but the distances between clusters mean little. Both panels above are hand-drawn schematics of typical results, not computed projections.

Compare the overlapping PCA clusters on the left with the separated t-SNE clusters on the right, then read the table. t-SNE plots need careful reading: cluster sizes and the gaps between clusters carry little meaning, and different random seeds give different layouts. It also has no transform for new points, so unlike PCA it cannot be fit on training data and applied to test data. UMAP (McInnes et al., 2018) is a faster alternative that keeps more global structure. A common recipe runs PCA down to about 50 dimensions first, then t-SNE.

What is dimensionality reduction?

Dimensionality reduction is the process of representing data that has many features with far fewer, while keeping as much of its structure as possible. Linear methods such as principal component analysis (PCA) project the data onto a few new axes. Non-linear methods such as t-SNE and UMAP place points in two or three dimensions so that neighbors stay neighbors.

It is used to speed up and regularize models, to compress data, to remove noise, and above all to visualize high-dimensional data. Biologists plot single-cell gene expression with t-SNE or UMAP, ML engineers inspect word and image embeddings the same way, and PCA is a standard preprocessing step before clustering or regression on wide tables. Reducing dimensions also eases the curse of dimensionality, where distances and density estimates break down as features are added.

Key ideas

Dimensionality reduction maps data with many features to fewer while keeping as much structure as possible.
In high dimensions data becomes sparse and distances lose contrast, which hurts many models.
PCA projects onto perpendicular directions of maximum variance, the eigenvectors of the covariance matrix.
The number of components can be chosen from the scree plot, a variance target, or cross-validation.
t-SNE keeps local neighborhoods for 2D visualization but distorts global distances.
PCA is linear, fast, and reusable on new data, while t-SNE is mainly a visualization tool.

Common questions

What is the difference between dimensionality reduction and feature selection?

Feature selection keeps a subset of the original columns unchanged. Dimensionality reduction methods such as PCA build new features that mix all the original ones. Selection keeps features interpretable, while reduction can pack more of the information into fewer numbers.

Is PCA supervised or unsupervised?

PCA is unsupervised: it ignores labels and only looks at the spread of the inputs. The direction with the most variance is not always the one that separates classes. Linear discriminant analysis (LDA) is the supervised counterpart, choosing directions that best separate known classes.

When does PCA fail?

PCA can only find flat subspaces, so it struggles when data lies on a curved surface, such as the classic swiss roll. It can also discard a low-variance direction that happens to predict the target. Kernel PCA, autoencoders, and manifold methods such as UMAP handle curved structure better.

Can t-SNE or UMAP be used as model input features?

UMAP can transform new points and is sometimes used that way, though results vary between runs. Standard t-SNE has no way to embed new points without rerunning on the whole dataset, so it is used almost only for visualization. For model inputs, PCA or a learned embedding is the safer choice.

Related explainers

Curse of Dimensionality — Why high-dimensional spaces break intuition and algorithms
Singular Value Decomposition — Factor any matrix into rotation, scaling, and rotation — the Swiss army knife of linear algebra
Eigenvalues & Eigenvectors — How matrices transform space — eigenvectors, eigenvalues, and decomposition
Feature Extraction — How networks find patterns in raw data
Autoencoders & VAEs — From compression to generation — standard and variational autoencoders
K-Means Clustering — Unsupervised grouping by centroid distance

References

Last updated October 25, 2026