ENID
Lensa ML

How Matrix Calculus Works

Step 1 of 6: Partial Derivatives

Hold one variable fixed, differentiate the other

x1.50
y1.00
xy∂f/∂x∂f/∂y(1.5, 1.0)
∂f/∂x = 2x
3.00
∂f/∂y = 2y
2.00
f = x² + y²
3.25

The surface is steeper in the x-direction (∂f/∂x = 3.0) than in y (∂f/∂y = 2.0). Moving along x changes f faster.

Move the x and y sliders and compare the two partial derivative cards. Together they define the best linear approximation of f near the point: f(x + Δx, y + Δy) ≈ f(x, y) + (∂f/∂x)Δx + (∂f/∂y)Δy. Matrix calculus writes this as f(x + Δx) ≈ f(x) + J·Δx, where J is a 1×2 row of partials. For a scalar function the Jacobian is one row, and the gradient is that row written as a column.

Step 2 of 6: The Gradient Vector

Combine partials into a vector that points uphill

x2.00
y1.00
∇f
∇f_x = 2x
4.00
∇f_y = 2y
2.00
|∇f| magnitude
4.47

The gradient is large (|∇f| = 4.5), so the surface is steep here. The arrow ∇f = [4.0, 2.0] points directly away from the center, because this bowl is perfectly round. On a stretched bowl the gradient would not point away from the minimum.

Drag the point and read off ∇f. In vector notation the bowl is f(x) = xᵀx, and its gradient is 2x, which is why the arrow is always twice the position vector. Two identities cover most ML derivations: the gradient of aᵀx is a, and the gradient of xᵀAx is (A + Aᵀ)x, or 2Ax when A is symmetric. Setting the gradient of the least-squares loss |Xw − y|² to zero with these rules gives the normal equations XᵀXw = Xᵀy.

Step 3 of 6: Gradient Points Uphill

The gradient always points toward steepest ascent

x1.50
y-1.00
Direction
-34°
Magnitude
3.61
f(x,y)
3.25

Every arrow points away from the center, uphill on the paraboloid. Negating the gradient gives the steepest descent direction, the step gradient descent takes.

Move the x and y sliders to place the bright arrow, then compare it with the grid. The whole field is itself a vector-valued function, mapping each point (x, y) to the vector (2x, 2y). Its Jacobian is the Hessian of f, the matrix of second partial derivatives. Here it is 2I. Curvature is equal in all directions, so the arrows grow at the same rate whichever way you move.

Step 4 of 6: Functions with Vector Outputs

Mapping grids through linear transformations

a1.00
b0.30
c-0.20
d1.00
J = [1.0, 0.3; -0.2, 1.0]
det(J)
1.06
Area scale
1.06×
Orientation
Kept

det(J) = 1.06 > 1: the map expands area. Each unit square becomes 1.06× larger. The matrix is called J because, for a linear map f(v) = Jv, its matrix of partial derivatives ∂fᵢ/∂xⱼ is J itself.

Adjust the four entries a, b, c, d and watch the 5×5 grid deform. The map sends (x, y) to (ax + by, cx + dy). Its Jacobian is the matrix [[a, b], [c, d]] itself. It does not depend on the point, so the grid lines stay straight and evenly spaced. A neural network layer y = Wx has Jacobian W. A nonlinear map, such as an elementwise activation, has a Jacobian that changes from point to point. For ReLU or tanh it is diagonal, since each output depends on only one input.

Step 5 of 6: The Jacobian Matrix

How a small patch of space gets stretched and rotated

a1.30
b0.40
c-0.30
d1.10
inputoutput
det(J)
1.55
Area ratio
1.55
Input area
1
Output area
1.55

The unit square (area = 1) maps to a parallelogram with area |det(J)| = 1.55. For a linear map this scale factor is the same at every point. In general J holds the partial derivatives J_ij = ∂fᵢ/∂xⱼ: for a linear map f(v) = Jv those are just the matrix entries, while for a nonlinear map J, and so |det J|, changes from point to point.

Change the four entries and compare the input and output area cards. For nonlinear maps the same rule holds locally: a tiny patch around a point has its area scaled by |det J| at that point. This is the change-of-variables formula behind normalizing flows, where the density of a transformed sample is p(x) / |det J|. Flow architectures such as RealNVP design layers with triangular Jacobians so that det J is a product of diagonal entries and cheap to compute.

Step 6 of 6: Jacobians in Backprop

Gradients flow backward through a chain of Jacobians

w₁ (layer 1 weight)0.80
w₂ (layer 2 weight)1.20
Forward signalBackward gradient×w₁×w₂1.000.800.96xw₁·xw₂·w₁·x×J₂=w₂×J₁=w₁0.961.201.00J_total = w₁ × w₂ = 0.96
w₁
0.80
w₂
1.20
J total (w₁×w₂)
0.96
Gradient
0.96

|w₁ × w₂| = 0.96 is close to 1, so the gradient passes through at about the same size, neither vanishing nor exploding.

Set w₁ and w₂ and watch the forward signal and backward gradient bars. Here each layer is scalar, so each Jacobian is 1×1 and the chain is plain multiplication. In a real network the total Jacobian is J₃·J₂·J₁, one matrix per layer. Backpropagation never forms it. It starts from the loss gradient, a row vector, and multiplies it by one Jacobian at a time from the right. These vector-Jacobian products cost about as much as the forward pass.

What is matrix calculus?

Matrix calculus is the notation and rules for differentiating functions whose inputs or outputs are vectors or matrices. A scalar function of a vector has a gradient. A vector-valued function has a Jacobian, the matrix of every output's partial derivative with respect to every input. The rules let you write these derivatives in a few symbols instead of one partial at a time.

Deep learning is written in vectors and matrices, so its derivatives are too. A layer y = Wx + b has a Jacobian with respect to x and a gradient with respect to W, and backpropagation multiplies these Jacobians layer by layer. Matrix calculus also appears in least squares, Gaussian likelihoods, change-of-variables formulas for normalizing flows, and second-order optimizers.

Key ideas

  • The Jacobian of f: ℝⁿ → ℝᵐ is an m×n matrix of partial derivatives.
  • For a scalar function the Jacobian is a single row, the transpose of the gradient.
  • A linear map x ↦ Ax has Jacobian A everywhere.
  • |det J| is the local factor by which a map scales area or volume.
  • The chain rule for vector functions multiplies Jacobians: J_{f∘g} = J_f · J_g.
  • Backpropagation computes vector-Jacobian products instead of full Jacobian matrices.

Common questions

What are numerator layout and denominator layout?

They are two conventions for arranging the same partial derivatives. Numerator layout makes ∂y/∂x an m×n matrix with one row per output; denominator layout uses the transpose. Textbooks and papers mix them, so many apparent sign or transpose errors are layout mismatches. Checking that matrix shapes line up is the quickest test.

What is the gradient of a loss with respect to a weight matrix?

For a layer y = Wx with upstream gradient g = ∂L/∂y, the gradient is ∂L/∂W = g·xᵀ, an outer product with the same shape as W. Over a batch these outer products are summed, which is why weight gradients are computed as one matrix multiplication of the upstream gradients with the layer inputs.

How is the Hessian related to the Jacobian?

The Hessian of a scalar function is the Jacobian of its gradient. For a model with n parameters it has n² entries, so it is almost never formed in deep learning. Methods that need curvature use Hessian-vector products instead, which autodiff computes without building the matrix.

When do you need the full Jacobian?

Rarely in training, since a scalar loss needs only vector-Jacobian products. Full Jacobians appear in normalizing flows (for the determinant), in robustness analysis, and in some physics and control models. Libraries such as JAX provide jacfwd and jacrev, which build them column by column or row by row.

Related explainers

References

  1. Petersen & Pedersen, "The Matrix Cookbook"
  2. Parr & Howard (2018), "The Matrix Calculus You Need For Deep Learning"
  3. Deisenroth, Faisal & Ong, "Mathematics for Machine Learning", ch. 5 Vector Calculus
  4. Dinh, Sohl-Dickstein & Bengio (2016), "Density estimation using Real NVP"

Last updated