ENID
Lensa ML

How Eigenvalues Work

Step 1 of 6: Linear Transformations

A matrix stretches, squashes, and flips space

a (diagonal)2.00
b (off-diagonal)0.50
d (diagonal)1.00
-4-3-2-11234-2-112A = [[2.0, 0.5], [0.5, 1.0]]
det(A)
1.75
Trace
3.00

The determinant is greater than 1, so this matrix expands area. The grid of points covers more of the plane.

Drag a, b, and d and watch the grid of points move. The matrix here is [[a, b], [b, d]], which is symmetric because the same b sits in both off-diagonal slots. At the default values, det = 2·1 − 0.5² = 1.75 and the trace is 3. Two facts link these cards to later steps: the trace equals λ₁ + λ₂, and the determinant equals λ₁·λ₂. So a negative determinant always means the eigenvalues have opposite signs.

Step 2 of 6: Special Directions

Eigenvectors only scale — they never rotate

a (diagonal)2.00
b (off-diagonal)0.50
d (diagonal)1.00
-4-3-2-11234-2-112v₁v₂
λ₁
2.207
λ₂
0.793

Eigenvectors are special directions that only scale under the matrix; they never rotate. The pink (v₁) and purple (v₂) arrows stay on the same line after transformation. Other vectors (gray) change direction. Change a, b, or d and the eigenvectors move. Every matrix here is symmetric (both off-diagonal entries are b), which guarantees real eigenvalues and perpendicular eigenvectors. A general matrix need not have either: a rotation has no real eigenvectors at all.

Adjust the matrix and watch which arrows stay on their own line. Symmetric matrices always have real eigenvalues and perpendicular eigenvectors; this is the spectral theorem. Non-symmetric matrices need not: a 90° rotation matrix moves every direction, so it has no real eigenvectors at all. Multiplying any starting vector by A over and over turns it toward the eigenvector with the largest |λ|. That method, power iteration, is how PageRank ranks web pages.

Step 3 of 6: Eigenvalues as Scale Factors

How much each eigenvector stretches or flips

a (diagonal)2.00
b (off-diagonal)0.50
d (diagonal)1.00
-4-3-2-11234-2-112scaled by λ₁=2.21scaled by λ₂=0.79
λ₁
2.207
λ₂
0.793
|λ₁|
2.207
|λ₂|
0.793

Both eigenvalues are positive, so the matrix stretches or compresses along both eigenvector directions without flipping either.

Change a, b, and d and compare λ₁ and λ₂ with their absolute values. At the defaults they are about 2.21 and 0.79. Applying A k times multiplies each eigenvector component by λᵏ, so directions with |λ| > 1 grow and those with |λ| < 1 decay. The largest |λ| is the spectral radius. A recurrent network multiplies its hidden state by the same weight matrix at every time step, so a spectral radius far from 1 leads to exploding or vanishing signals.

Step 4 of 6: The Characteristic Equation

Finding eigenvalues by solving det(A - λI) = 0

a (diagonal)2.00
b (off-diagonal)0.50
d (diagonal)1.00
λλ1=2.21λ2=0.79det(A - λI) = λ² - 3.0λ + 1.75 = 0
Trace (a+d)
3.00
det (ad-b²)
1.75
λ₁
2.207
λ₂
0.793

Moderate discriminant: two distinct real eigenvalues, where the parabola crosses zero at two separate points.

Move the sliders and watch the parabola's roots, which are the eigenvalues. For a 2×2 matrix, det(A − λI) = λ² − (trace)·λ + det. Its discriminant here is (a − d)² + 4b², which can never be negative, so a symmetric matrix never has complex eigenvalues. For n×n matrices the polynomial has degree n, and polynomials of degree 5 or more have no general root formula. Software therefore uses iterative methods such as the QR algorithm instead.

Step 5 of 6: Eigendecomposition

A = PDP⁻¹ — rotate, scale, rotate back

a (diagonal)2.00
b (off-diagonal)0.50
d (diagonal)1.00
Decomposition phase0.00
-4-3-2-11234-2-112A = PDP⁻¹
Phase
P⁻¹ (rotate)
λ₁
2.207
λ₂
0.793

Phase 1: P⁻¹ rotates the plane so the eigenvectors line up with the axes, the matrix's own coordinate system. A rotated circle is the same circle, so the outline does not change in this phase.

Drag the Decomposition Phase slider from 0 to 3 to watch the unit circle become an ellipse in three moves. Because this matrix is symmetric, P is orthogonal and P⁻¹ = Pᵀ, so the first and last moves are pure rotations. The decomposition makes powers easy: Aᵏ = P·Dᵏ·P⁻¹, and Dᵏ just raises each eigenvalue to the k-th power. A non-symmetric matrix may lack enough eigenvectors to build P; the SVD always works instead.

Step 6 of 6: Why Eigenvectors Matter

Principal Component Analysis finds the directions of most variance

Spread ratio2.00
Rotation angle (°)30
-4-3-2-11234-2-112PC1PC2
PC1 variance
1.22
PC2 variance
0.17
Total var
1.39
Explained %
88%

One direction dominates, and PC1 carries most of the variance. Projecting onto PC1 alone keeps most of the information.

Raise the Spread ratio, then change the Rotation angle. Rotating the cloud moves the principal component arrows but leaves their variances unchanged, because eigenvalues describe the shape of the data, not its orientation. The 30 points give a 2×2 covariance matrix, and its eigenvalues are the variances along each component. Analysts often keep enough components to explain 90 to 95 percent of the total variance, which is the sum of the eigenvalues.

What are eigenvalues and eigenvectors?

An eigenvector of a square matrix A is a nonzero vector whose direction does not change when A is applied: Av = λv. The number λ is its eigenvalue, the factor by which that vector is stretched, shrunk, or flipped. Together they describe what a matrix does along its own natural axes.

In machine learning, the eigenvectors of a covariance matrix are the principal components used in PCA. Eigenvalues of the Hessian describe the curvature of a loss surface and limit the learning rate gradient descent can use. They also govern whether repeated matrix multiplication blows up or dies out, which matters for recurrent networks, and the eigenvalues of a graph's Laplacian drive spectral clustering.

Key ideas

  • An eigenvector keeps its direction under a matrix: Av = λv.
  • The eigenvalue λ is the scale factor along that eigenvector, and a negative λ flips it.
  • The trace is the sum of the eigenvalues, and the determinant is their product.
  • Eigenvalues are the roots of the characteristic equation det(A − λI) = 0.
  • Symmetric matrices have real eigenvalues and perpendicular eigenvectors.
  • In PCA, the eigenvectors of the covariance matrix are the principal components.

Common questions

What do the eigenvalues of the Hessian tell you about training?

Each eigenvalue is the curvature of the loss along its eigenvector. At a critical point, all positive means a local minimum and mixed signs mean a saddle. For gradient descent on a quadratic, the learning rate must stay below 2/λ_max to avoid divergence, so the sharpest direction sets the speed limit.

Can eigenvalues be complex numbers?

Yes, for non-symmetric real matrices. A rotation by θ in the plane has eigenvalues cos θ ± i·sin θ, which signal turning rather than stretching. Complex eigenvalues always come in conjugate pairs for real matrices, and their magnitude still decides growth or decay.

What is a positive definite matrix?

A symmetric matrix whose eigenvalues are all positive, which is the same as xᵀAx > 0 for every nonzero x. Covariance matrices are positive semidefinite, with eigenvalues of zero or more. A positive definite Hessian at a critical point means the point is a strict local minimum.

Should I use numpy.linalg.eig or eigh?

Use eigh for symmetric or Hermitian matrices such as covariance matrices. It is faster, returns real eigenvalues in ascending order, and guarantees orthonormal eigenvectors. eig handles general square matrices but may return complex values, even from round-off on a matrix that should be symmetric.

Related explainers

References

  1. Strang, MIT 18.06 Linear Algebra, lecture 21 Eigenvalues and Eigenvectors
  2. Deisenroth, Faisal & Ong, "Mathematics for Machine Learning", ch. 4 Matrix Decompositions
  3. Pearson (1901), "On lines and planes of closest fit to systems of points in space"

Last updated