ENID
Lensa ML

Linear Algebra for ML

Step 1 of 6: What's a Vector?

A list of numbers with a length and a direction

v
x3.0
y2.0
scale c1.0
Vector c·v
[3.0, 2.0]
Length |c·v|
3.61
Direction
34°

A vector is an ordered list of numbers, here [3.0, 2.0]: 3.0 along x and 2.0 along y. Drawn as an arrow it has a length, √(x² + y²) = 3.61, and a direction. Drag the scale to stretch it. The same rules work for lists of 768 numbers, which is how a language model stores one token.

Drag the components and the scale. A vector in n dimensions is an ordered list of n numbers; its length is √(x₁² + … + xₙ²). Adding vectors adds their components, and multiplying by a number scales every component, which stretches the arrow or, for a negative number, flips it. Language models represent each token as a vector of hundreds or thousands of numbers, and everything they compute starts from operations like these.

Step 2 of 6: Vectors as Data

Differences and distances between data points

height (m) →↑ weight (100 kg)Ana [1.62, 0.55]Budi [1.78, 0.82]Citra [1.55, 0.48]Dewi [1.70, 0.62]Eko [1.85, 0.90]Fajar [1.66, 0.74]

Buttons pick the first person; click a dot to pick the second.

Eko − Ana
[0.23, 0.35]
Distance |b − a|
0.419
Closest to Ana
Citra

Each person is a vector of measurements: Ana is [1.62, 0.55]. Once data are vectors, geometry becomes a tool. Subtracting gives the arrow from Ana to Eko, [0.23, 0.35], and its length, 0.419, is how different they are. The nearest neighbour of Ana is Citra, the idea behind k-NN and vector search. Units matter: weight is in 100 kg so both axes span similar ranges; in kilograms, weight would swamp height, which is why features get scaled.

Pick two people and compare them. A dataset with m examples and n features is an m × n matrix, one row per example. The Euclidean distance between two rows is the length of their difference vector, and it is what k-nearest neighbours and k-means use. Because distance adds up squared differences, a feature measured in large units dominates, which is why features are usually standardized before distance-based methods.

Step 3 of 6: The Dot Product

Multiply, add, and read it as a shadow

ab
a·b = 2.82×0.86 + 1.03×2.35 = 4.82
|a|3.0
angle a20°
|b|2.5
angle b70°
a·b
4.82
|a||b|cos θ
4.82
cos θ
0.64
Shadow of b on a
1.61

Multiply matching components and add: 4.82. Geometrically that equals |a|·|b|·cos θ, the length of a times the length of b's shadow on a (the amber bar). They point roughly the same way, so it is positive. Note it grows with length: double |a| and a·b doubles while the angle stays put. To compare only directions, divide by both lengths to get the cosine, 0.64; that's cosine similarity, used for embeddings.

Change the lengths and angles of a and b. The dot product a · b = Σ aᵢbᵢ = |a||b| cos θ: positive when the vectors point the same way, zero when they are perpendicular, negative when they oppose. Dividing by both lengths gives cosine similarity, which ignores length; it is the standard score for comparing text embeddings in search and retrieval. Every neuron computes a dot product between its weights and its input.

Step 4 of 6: What's a Matrix?

Its columns say where the axes land

ê₁ →ê₂ →
1.001.000.001.00
a1.0
c0.0
b1.0
d1.0
ê₁ = [1,0] lands at
[1.00, 0.00]
ê₂ = [0,1] lands at
[1.00, 1.00]
det = area scale
1.00

A 2×2 matrix is a function that moves every point of the plane. The trick to reading it: the first column [1.00, 0.00] is where ê₁ = [1, 0] lands, and the second column [1.00, 1.00] is where ê₂ = [0, 1] lands. Every other point follows, because [x, y] = x·ê₁ + y·ê₂, so grid lines stay straight, parallel and evenly spaced. The determinant, 1.00, is how much areas grow: the unit square becomes the amber shape with area 1.00.

Try the presets, then drag the four entries. A matrix is a linear transformation: it keeps the origin fixed and keeps grid lines straight, parallel, and evenly spaced. Its columns are the images of the basis vectors, which is how you read any matrix at a glance. The determinant is the factor by which areas scale; it is zero exactly when the matrix squashes space into a lower dimension and has no inverse.

Step 5 of 6: Matrix Multiplication

One matrix for two moves, and AB ≠ BA

dashed grey = original F · dotted = after the first matrix · filled = after both · thin outline = the other order

AB =
entry (1,2) = row 1 of the left matrix · column 2 of the right = 0.87×1.00 + -0.50×1.00 = 0.37
AB equals BA?
no
det(AB) = det A · det B
1.00 = 1 × 1
F's top corner [0,2] lands at, AB vs BA
[0.7, 2.7] vs [0.7, 1.7]

Multiplying two matrices gives one matrix that does both moves. AB applies the right-hand matrix first (shear, the dotted shape) and then the left one (rotate, the filled shape), like function composition. Each entry of the product is a row of the left matrix dotted with a column of the right: click an entry to see it. Order matters: the thin outline is the other order, and the two shapes differ, so AB ≠ BA. That's why a network applies its layers in a fixed order.

Switch the order and click entries of the product. The product AB is the single matrix that applies B first and then A, and entry (i, j) is row i of A dotted with column j of B. It is defined only when the inner sizes match, (m × k)(k × n) gives m × n, and in general AB ≠ BA. Determinants multiply, det(AB) = det(A)·det(B), so composing two area-preserving maps preserves area.

Step 6 of 6: Why It Matters for ML

A dense layer is ReLU(Wx + b)

1.0x10.5x20.65y1 = ReLU(0.65)0.55y2 = ReLU(0.55)0.00y3 = ReLU(-0.20)y1 = ReLU(0.8×1.0 + -0.5×0.5 + 0.1) = 0.65

Click an output to see its row of W.

x11.0
x20.5
W shape · x shape
(3, 2) · (2,)
Wx + b
[0.7, 0.6, -0.2]
After ReLU
[0.7, 0.6, 0.0]

A dense layer is exactly this: y = ReLU(Wx + b). Each output neuron's value is one row of W dotted with the input, the step-3 dot product, plus its own bias: y1 uses row [0.8, -0.5]. The thick green and red lines are those weights. Computing all three at once is a matrix-vector product, and a batch of inputs makes it a matrix-matrix product, which is what GPUs do trillions of times per second. Without the ReLU, stacking layers would just multiply matrices together into one, step 5, so the non-linearity is what makes depth worth having.

Move the inputs and click each output. A dense layer computes y = σ(Wx + b): a matrix-vector product, a bias, and an element-wise nonlinearity. For a batch, the inputs become a matrix X and the layer computes XWᵀ + b in one matrix multiplication, which is why GPU throughput is measured in multiply-adds per second. Attention, convolution, and embeddings are all built from the same matrix operations.

What is linear algebra in machine learning?

Linear algebra is the mathematics of vectors, matrices, and the linear transformations between them. A vector is an ordered list of numbers, a matrix is a grid of numbers, and multiplying a vector by a matrix maps it to a new vector. The rules are simple and the same in two dimensions or two million.

Machine learning stores almost everything as vectors and matrices: a dataset is a matrix with one row per example, a word embedding is a vector, and a layer's weights are a matrix. Training and inference mostly consist of matrix multiplications, which is why GPUs, built to do many multiply-adds in parallel, made modern deep learning practical. PCA, linear regression, attention, and recommender systems are all linear algebra at their core.

Key ideas

  • A vector is an ordered list of numbers that can be read as a point or an arrow.
  • Each row of a dataset is a vector, and the whole dataset is a matrix.
  • The dot product a · b = |a||b|cos(θ) measures how aligned two vectors are.
  • A matrix is a linear transformation, and its columns show where the basis vectors land.
  • Each entry of a matrix product is a row-by-column dot product.
  • A dense neural network layer is a matrix-vector product followed by a nonlinear activation.

Common questions

How much linear algebra do you need for machine learning?

For applying standard models, vectors, matrices, dot products, matrix multiplication, and transposes cover most needs. Reading research or deriving methods also needs eigenvalues, SVD, matrix calculus, and projections. A course such as Strang's MIT 18.06 covers all of this.

What is the difference between a vector, a matrix, and a tensor?

A vector has one index, a matrix has two, and a tensor is the general name for an array with any number of indices. A batch of RGB images is a 4-dimensional tensor: batch, channels, height, width. Deep learning libraries call every such array a tensor.

What is the transpose of a matrix?

The transpose Aᵀ swaps rows and columns, so entry (i, j) becomes entry (j, i). It turns a column vector into a row vector, which lets the dot product be written as aᵀb. In backpropagation, the gradient flowing back through a layer with weights W is multiplied by Wᵀ.

What does it mean for vectors to be linearly independent?

No vector in the set can be written as a combination of the others. In a dataset, a column that is an exact combination of others, such as a total that equals the sum of two other columns, makes the columns dependent. That makes XᵀX singular and breaks the closed-form solution for linear regression.

Related explainers

References

  1. Strang, MIT 18.06 Linear Algebra, MIT OpenCourseWare
  2. Deisenroth, Faisal & Ong, "Mathematics for Machine Learning", ch. 2 Linear Algebra
  3. Goodfellow, Bengio & Courville, "Deep Learning", ch. 2 Linear Algebra

Last updated