How Matrix Calculus Works
Partial Derivatives
Hold one variable fixed, differentiate the other
The surface is steeper in the x-direction (∂f/∂x = 3.0) than in y (∂f/∂y = 2.0). Moving along x changes f faster.
Step 1 of 6: Partial Derivatives
Hold one variable fixed, differentiate the other
The surface is steeper in the x-direction (∂f/∂x = 3.0) than in y (∂f/∂y = 2.0). Moving along x changes f faster.
Step 2 of 6: The Gradient Vector
Combine partials into a vector that points uphill
The gradient is large (|∇f| = 4.5) — the surface is very steep here. The arrow ∇f = [4.0, 2.0] points directly away from the center.
Step 3 of 6: Gradient Points Uphill
The gradient always points toward steepest ascent
Every arrow points away from the center (uphill on the paraboloid). Negate the gradient to get the steepest descent direction — that's exactly what gradient descent does!
Step 4 of 6: Functions with Vector Outputs
Mapping grids through linear transformations
det(J) = 1.06 > 1 — the transformation expands area. Each unit square becomes 1.06× larger.
Step 5 of 6: The Jacobian Matrix
How a small patch of space gets stretched and rotated
The unit square (area = 1) maps to a parallelogram with area |det(J)| = 1.55. The Jacobian determinant is the local area scaling factor at every point.
Step 6 of 6: Jacobians in Backprop
Gradients flow backward through a chain of Jacobians
|w₁ × w₂| = 0.96 ≈ 1 — the gradient is stable! This is the sweet spot where learning signals propagate without vanishing or exploding.