1Linear Algebra 2Tensors, Broadcasting & Reshaping 3Norms & Distance Metrics 4Eigenvalues & Eigenvectors 5Singular Value Decomposition

Lensa ML

How Norms & Distances Work

Step 1 of 6: The L2 (Euclidean) Norm

Straight-line distance from the origin

x3.00

y2.00

3.00

2.00

L2 norm

3.606

This vector is far from the origin, and its L2 norm is that straight-line distance.

Drag the x and y sliders between −4 and 4. At the starting point (3, 2) the length is √13 ≈ 3.61. In n dimensions the same formula sums n squares: the vector (1, 1, …, 1) with 100 entries has L2 norm 10. ML code often uses the squared norm |x|² instead, because it avoids the square root and has the simple gradient 2x. Weight decay and mean squared error are both built on this squared form.

Step 2 of 6: The L1 (Manhattan) Norm

Distance measured along axes — the taxicab metric

x3.00

y2.00

L1 norm

5.000

L2 norm

3.606

L1 / L2 ratio

1.387

The L1 norm (5.00) is significantly larger than L2 (3.61). The taxi follows the grid lines while the straight line cuts across. Off the axes, the gap is largest at 45°, where L1 = √2·L2.

Move x and y and watch the L1 / L2 RATIO card. At (3, 2) the L1 norm is 5 and the ratio is about 1.39. In n dimensions the ratio can reach √n, so the gap between the two norms grows with dimension. L1 is less sensitive to outliers: one error of 10 adds 10 to an L1 loss but 100 to a squared L2 loss. That is why mean absolute error is the more robust regression loss. L1 has a kink at zero, so optimizers use its subgradient there.

Step 3 of 6: Unit Balls & Lp Norms

Watch the unit ball morph as p changes

p (norm order)2.00

p value

2.0

Shape

Circle

Area

3.142

Near p = 2 the unit ball is a circle, the Euclidean norm. It treats all directions equally, so it is rotationally invariant.

Slide p (norm order) from 0.5 to 10 and watch the AREA card. The general formula is |x|_p = (Σ|xᵢ|^p)^(1/p), and the unit ball is every point with norm 1. Its area goes from 2 at p = 1, to π at p = 2, toward 4 as p grows. For p ≥ 1 the triangle inequality holds and the ball is convex. Values below 1 break it, so they are not true norms. The count of nonzero entries, sometimes called L0, is the extreme case.

Step 4 of 6: Distance Between Vectors

L1, L2, and L∞ — three ways to measure closeness

x₁-2.00

y₁1.00

x₂2.00

y₂3.00

L1 dist

6.000

L2 dist

4.472

L∞ dist

4.000

Three ways to measure the same gap: L1 (6.00) walks the grid, L2 (4.47) flies straight, and L∞ (4.00) only tracks the dominant axis. L∞ ≤ L2 ≤ L1 always holds.

Move both points with the x₁, y₁, x₂, y₂ sliders. At the start, (−2, 1) and (2, 3) differ by (4, 2), so the distances are 6 for L1, about 4.47 for L2, and 4 for L∞. L∞ is also called Chebyshev distance: the number of moves a chess king needs. Adversarial robustness research often bounds attacks in L∞, for example allowing each pixel to change by at most 8/255. The choice of metric also changes which neighbors k-NN finds.

Step 5 of 6: Cosine Similarity

Comparing direction, not magnitude

Angle of vector 1 (°)30

Angle of vector 2 (°)120

Angle

90°

cos(θ)

0.000

Dot product

0.000

The vectors are nearly perpendicular, so cosine similarity ≈ 0. In an embedding space this usually means the items are unrelated. Both vectors here have length 1, which is the only reason the dot product equals cos θ. Scale a vector up and its dot product grows, while its cosine similarity stays the same.

Set Angle₁ and Angle₂ and compare cos(θ) with the dot product. Both vectors have length 1 here, so the two cards always match. For vectors of any length, cos(θ) = a·b / (|a||b|). Embedding models are usually trained so that cosine similarity reflects meaning, and many vector databases store normalized vectors so a plain dot product gives the cosine. The starting angles, 30° and 120°, are 90° apart, which gives a similarity of exactly 0.

Step 6 of 6: Norms in ML

How L1 and L2 regularization constrain weights

w₁1.50

w₂2.00

λ (regularization strength)0.50

L1 penalty

1.750

L2 penalty

3.125

w₁

1.50

w₂

2.00

At λ = 0.50, the constraints tighten. The L1 diamond has corners on the axes, and solutions tend to land there, zeroing out a weight. L2's circle has no corners. The dashed budget is drawn with radius 1/λ for illustration: a bigger λ means a smaller budget, but the exact match depends on the data.

Set w₁ and w₂, then raise λ (regularization strength). At the defaults (1.5, 2.0) with λ = 0.5, the L1 penalty is 0.5 × 3.5 = 1.75 and the L2 penalty is 0.5 × 6.25 = 3.125. The gradients explain the different behavior. L1's gradient is λ·sign(w), the same size for every weight, so small weights get pushed all the way to zero. L2's gradient is 2λw, which fades as w shrinks. Elastic net combines both penalties.

What are vector norms and distance metrics?

A norm is a function that assigns a length to a vector. The L2 norm is ordinary straight-line length, √(x₁² + x₂² + …); the L1 norm adds up absolute values; and the L∞ norm takes the largest absolute value. Every norm defines a distance: the distance between a and b is the norm of a − b.

Machine learning uses norms and distances wherever it compares or limits vectors. k-nearest neighbors and k-means rely on distances between data points, retrieval systems rank embeddings by cosine similarity, and L1 and L2 penalties keep model weights small. Loss functions are norms too: mean squared error is based on L2 and mean absolute error on L1.

Key ideas

A norm measures the length of a vector, and the distance between two vectors is the norm of their difference.
L2 is straight-line length, L1 sums absolute values, and L∞ takes the largest absolute value.
For any vector, L∞ ≤ L2 ≤ L1.
The shape of the unit ball changes with p, from a diamond at p = 1 to a circle at p = 2 to a square as p → ∞.
Cosine similarity compares direction only and ignores length.
L1 regularization produces sparse weights, while L2 shrinks all weights without zeroing them.

Short takes on this topic

Cosine vs Euclidean: why embeddings compare angles

Common questions

Should I use cosine similarity or Euclidean distance?

Use cosine when only direction matters, as with text embeddings whose lengths vary for reasons unrelated to meaning. For unit-length vectors the two agree in ranking, because |a − b|² = 2 − 2·cos(θ). Euclidean distance is the better choice when magnitude carries information, such as physical measurements.

Why do distances become less useful in high dimensions?

As the number of dimensions grows, the distances from one point to all others tend to bunch together, so the nearest and farthest neighbors are almost equally far. Methods that rely on distance, such as k-NN, then lose contrast. Dimensionality reduction, feature selection, or learned embeddings usually help.

What is the Mahalanobis distance?

It is a distance that accounts for the scale and correlation of the data: d(x, y) = √((x − y)ᵀ Σ⁻¹ (x − y)), where Σ is the covariance matrix. If the features are uncorrelated with unit variance, it reduces to Euclidean distance. It is used in outlier detection and in Gaussian models.

Do features need to be scaled before computing distances?

Usually yes. A feature measured in thousands, such as income, will dominate one measured in single digits, such as rooms per house, in any L1 or L2 distance. Standardizing each feature to zero mean and unit variance gives each one an equal starting influence.

Related explainers

Linear Algebra — Vectors, matrices, and the language of data
Regularization — Dropout, L1, and L2 — keeping models honest
K-Nearest Neighbors — Classification by proximity and distance metrics
Embeddings — Turning words and categories into vectors
Curse of Dimensionality — Why high-dimensional spaces break intuition and algorithms
Feature Scaling — Why and how to normalize features for faster convergence

References

Last updated October 25, 2026