Singular Value Decomposition
What SVD Does
Any matrix = rotate, scale, rotate — decomposed into UΣVᵀ
The unit circle (dashed) transforms into an ellipse. SVD reveals that ANY matrix = rotate (Vᵀ) → scale (Σ) → rotate (U). The pink and purple arrows show the principal axes.
Step 1 of 6: What SVD Does
Any matrix = rotate, scale, rotate — decomposed into UΣVᵀ
The unit circle (dashed) transforms into an ellipse. SVD reveals that ANY matrix = rotate (Vᵀ) → scale (Σ) → rotate (U). The pink and purple arrows show the principal axes.
Step 2 of 6: U, Σ, V Separately
Watch each component's role: Vᵀ rotates, Σ scales, U rotates
Phase 1: Vᵀ rotates the input into the 'natural basis' of the matrix. The circle stays circular — only the coordinate system changes.
Step 3 of 6: Singular Values
σ₁ and σ₂ control how much the matrix stretches each axis
σ₁ is 3.0× larger than σ₂. The matrix amplifies one direction more than the other. The condition number (σ₁/σ₂) measures how 'skewed' the transformation is.
Step 4 of 6: Low-Rank Approximation
Keep the top-k singular values, discard the rest
Rank 5 captures 95.8% of the matrix's energy with only 80 parameters (vs 64 full). The discarded singular values were small — little information lost.
Step 5 of 6: SVD and PCA
SVD of centered data gives you principal components for free
With both components, reconstruction is perfect. PCA via SVD: center the data, compute SVD of the data matrix, and the right singular vectors ARE the principal components.
Step 6 of 6: SVD in ML
LoRA, compression, and recommender systems all rely on low-rank structure
4× compression — rank 16 uses only 2048 parameters vs 4096 full. This is the sweet spot for LoRA fine-tuning: enough rank to capture task-specific changes, huge memory savings.