ENID
Lensa ML

Singular Value Decomposition

Step 1 of 6: What SVD Does

Any matrix = rotate, scale, rotate — decomposed into UΣVᵀ

A[1,2] matrix element1.00
v₁v₂A = UΣVᵀ
σ₁
2.558
σ₂
0.977
σ₁/σ₂
2.62

The unit circle (dashed) becomes an ellipse. The pink and purple arrows mark its principal axes.

Drag the matrix element and watch the unit circle become an ellipse. The ellipse's two semi-axes have lengths σ₁ and σ₂, and their directions are the columns of U. This picture holds for every matrix, which is what makes SVD more general than eigendecomposition: a non-symmetric matrix can have complex eigenvectors or too few of them, but its singular vectors always exist and are always perpendicular. In three dimensions the circle becomes a sphere and the ellipse an ellipsoid, with one axis per singular value.

Step 2 of 6: U, Σ, V Separately

Watch each component's role: Vᵀ rotates, Σ scales, U rotates

Decomposition phase0.00
A = U · Σ · VᵀVᵀ (Rotate)
Phase
Vᵀ (Rotate)
σ₁
2.558
σ₂
0.977

Phase 1: Vᵀ rotates the input into the matrix's 'natural basis'. A rotated circle is the same circle, so the outline does not change in this phase.

Move the phase slider slowly from start to end. The first and last phases change only the circle's orientation, never its shape, because U and V are orthogonal: they preserve lengths and angles. All of the stretching happens in Σ. That separation makes inversion easy. Undo each piece in reverse order (rotate back by U, divide by each nonzero σ, rotate back by V) and you get the pseudoinverse, which least-squares solvers use when a matrix is not square or not invertible.

Step 3 of 6: Singular Values

σ₁ and σ₂ control how much the matrix stretches each axis

σ₁/σ₂ ratio3.00
σ₁=2.0σ₂=0.67
σ₁
2.00
σ₂
0.67
Ratio σ₁/σ₂
3.0
Area ratio
1.33

σ₁ is 3.0× larger than σ₂. The matrix amplifies one direction much more than the other.

Raise the σ₁/σ₂ ratio and watch the area ratio. For a 2×2 matrix, areas scale by σ₁·σ₂, which equals the absolute value of the determinant. The ratio σ₁/σ₂ is the condition number: when it is large, small errors in the input can become large errors in the output, so solving equations with that matrix is numerically fragile. In deep learning, σ₁ of a weight matrix bounds how much a layer can stretch its input, and spectral normalization divides each weight matrix by it to keep GAN training stable.

Step 4 of 6: Low-Rank Approximation

Keep the top-k singular values, discard the rest

Rank k (of 8)5
8.0σ14.0σ22.7σ32.0σ41.6σ51.3σ61.1σ71.0σ8Keep top-5 singular values95.8% energy retained
Rank k
5
Energy %
95.8%
Params (full)
64
Params (k)
80

Rank 5 captures 95.8% of the matrix's energy, but storing 5 columns of U and of V takes 80 numbers, more than the 64 of the full 8×8 matrix. Low rank k only saves space when k(m + n) < mn, here k < 4. On a 1000×1000 matrix, rank 10 needs 20,000 numbers instead of 1,000,000.

Lower the rank k and watch the parameter count drop. The Eckart–Young theorem says that keeping the top k singular values gives the best possible rank-k approximation: no other rank-k matrix is closer to the original. The energy readout is the share of the sum of squared singular values you keep. An m×n matrix stored at rank k needs k(m + n) numbers, with each σ folded into U or V, instead of mn, which is a large saving when k is small.

Step 5 of 6: SVD and PCA

SVD of centered data gives you principal components for free

Number of components2
PC1PC2SVD of centered data = PCA
Components
2
Explained
100.0%
σ₁²/N
1.16
σ₂²/N
0.09

With both components, reconstruction is perfect: two components span the whole 2D plane. Each PC's variance is σᵢ²/N, where σᵢ is a singular value of the centered N-point data matrix.

Switch between one and two components. If the data matrix X is centered, so every column has mean zero, its right singular vectors are the principal directions, and σᵢ²/n is the variance along each one, or σᵢ²/(n − 1) for the unbiased sample estimate. Libraries such as scikit-learn compute PCA this way rather than forming the covariance matrix XᵀX, because forming XᵀX squares the condition number and loses precision.

Step 6 of 6: SVD in ML

LoRA, compression, and recommender systems all rely on low-rank structure

Compression ratio (d/r)4
ΔW (64×64)4096 params≈B (64×16)·A (16×64)2048 params (50% saved)
Full params
4,096
LoRA params
2,048
Rank r
16
Savings
50%

4× compression: rank 16 uses only 2048 parameters vs 4096 full, enough rank for many task-specific changes. LoRA keeps the pretrained weights W₀ frozen and learns only the update: W = W₀ + ΔW, with ΔW = B·A.

Change the rank r and compare the full and LoRA parameter counts. LoRA does not run an SVD. It trains two small matrices whose product is the weight update, and SVD explains why that can work: if the update has most of its energy in a few singular directions, a rank-r product can capture it. Recommender systems use the same idea by factoring a user-by-item ratings matrix into user and item vectors, and latent semantic analysis factors a term-by-document matrix to find topics.

What is singular value decomposition (SVD)?

Singular value decomposition factors any real matrix A into three pieces, A = UΣVᵀ. U and V are orthogonal matrices, which only rotate or reflect, and Σ is a diagonal matrix of non-negative singular values sorted from largest to smallest. Every matrix has an SVD, including rectangular ones, which is not true of eigendecomposition.

In machine learning, SVD is behind principal component analysis (PCA), low-rank compression, recommender systems, latent semantic analysis, and the reasoning behind LoRA fine-tuning. The singular values also tell you a matrix's rank and how numerically stable it is to work with.

Key ideas

  • Every matrix factors as A = UΣVᵀ: rotate, scale along the axes, rotate.
  • Singular values are non-negative, sorted, and unique for a given matrix.
  • The number of nonzero singular values is the rank.
  • Keeping the top k singular values gives the best rank-k approximation.
  • PCA on centered data is an SVD of the data matrix.
  • Low-rank structure is what makes compression, recommenders, and LoRA work.

Common questions

Are singular values the same as eigenvalues?

Not in general. The singular values of A are the square roots of the eigenvalues of AᵀA. They match the eigenvalues only for symmetric matrices with non-negative eigenvalues. Eigenvalues can be negative or complex; singular values are always real and non-negative.

What does a zero singular value mean?

The matrix flattens at least one direction completely, so it is not full rank and cannot be inverted. The right singular vectors that go with zero singular values span the null space: the inputs the matrix sends to zero.

Is the SVD unique?

The singular values are. The singular vectors are unique up to a sign flip when the singular values are all different. If two singular values are equal, any rotation of their vectors within that plane is also a valid SVD.

How is SVD computed in practice?

Libraries such as NumPy and PyTorch call LAPACK routines, which reduce the matrix to a simpler form and then iterate. For an m×n matrix with m ≥ n the cost is O(mn²). For very large matrices where only the top k values are needed, randomized SVD computes them much faster.

Related explainers

References

  1. Eckart & Young (1936), "The approximation of one matrix by another of lower rank", Psychometrika
  2. Halko, Martinsson & Tropp (2011), "Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions"
  3. Miyato et al. (2018), "Spectral Normalization for Generative Adversarial Networks"
  4. Hu et al. (2021), "LoRA: Low-Rank Adaptation of Large Language Models"

Last updated