ENID
Lensa ML

Partial Derivatives & Gradients

Step 1 of 6: From 1D to 2D

Single derivatives become partial derivatives in multiple dimensions

Blend: 1D → 2D0.00
f(x) = x²x12468f(x,y) = x² + y²xy
Dimensions
1D
Variables
x
Derivative
df/dx

In 1D, f(x) = x² has one derivative: df/dx = 2x. There is only one direction to move, so one slope describes it.

Drag the Blend slider from 1D to 2D and watch the parabola turn into circular contours. Each ring is a level set: every point on it has the same value of f(x, y) = x² + y². A one-variable function has a single slope at each point, but a surface has a slope in every direction you could walk. Partial derivatives pick out two of those directions, along the x-axis and along the y-axis, and from them every other directional slope can be computed.

Step 2 of 6: Partial Derivative Notation

Hold one variable fixed, differentiate the other

x value1.50
Slice: f(x, y=1) = x² + x·1 + 1²x
f(x,y)
4.75
∂f/∂x
4.00
x
1.50
y (fixed)
1.0

∂f/∂x = 4.00: f is increasing in x at this point, so the teal tangent tilts upward. Only x varies; y stays at 1.

Move the x value slider while y stays at 1. The curve on screen is one slice of f(x, y) = x² + xy + y², namely x² + x + 1. To differentiate with respect to x, treat y as a constant: ∂f/∂x = 2x + y. At x = 1.5 that gives 4. Doing the same for y gives ∂f/∂y = x + 2y, which is 3.5 at the same point. The curly ∂ signals that other variables exist and are frozen, unlike the d in df/dx.

Step 3 of 6: The Gradient Vector

Combine all partial derivatives into one vector: ∇f

x position1.50
y position1.00
∇f = (∂f/∂x, ∂f/∂y) = (2x, 2y)∇f
∂f/∂x
3.00
∂f/∂y
2.00
|∇f|
3.61
f(x,y)
3.25

Far from the minimum the bowl is steep, so the gradient is large. Here |∇f| = 2r, twice the distance from the origin.

Use the x position and y position sliders to place the point. For this bowl, ∇f = (2x, 2y), so at (1.5, 1) the gradient is (3, 2) with length √13 ≈ 3.61. The gradient is always perpendicular to the contour through the point, because moving along a contour leaves f unchanged. The slope in any unit direction u is the dot product ∇f · u, called the directional derivative. It is largest when u points along ∇f.

Step 4 of 6: Gradient Direction & Magnitude

The gradient points uphill with magnitude equal to the steepness

x position1.00
y position1.50
f(x,y) = sin(x)·cos(y)
|∇f|
0.840
Angle (°)
-87.4
f(x,y)
0.060

Strong gradient (|∇f| = 0.84) pointing at -87°. The surface is steep here. Moving opposite to the arrow is the fastest way downhill, which is what gradient descent does.

Drag the point around the sin(x)·cos(y) surface and watch the angle readout. Here ∂f/∂x = cos(x)·cos(y) and ∂f/∂y = −sin(x)·sin(y). The peak at (π/2, 0) and the valley at (−π/2, 0) both have zero gradient, and so does the saddle at (0, π/2), where f rises in one direction and falls in another. A zero gradient alone cannot tell these apart; the second partial derivatives, collected in the Hessian, can.

Step 5 of 6: Gradient Descent Connection

Step opposite to ∇f to minimize the loss

Learning rate0.15
Step0
f(x,y) = x² + 3y²min
Step
0
f(x,y)
18.250
x
2.500
y
2.000

Starting point (2.5, 2.0), far from the minimum. The teal arrow shows −∇f, the direction of the first step. Increase the step slider to descend.

Set the Learning rate, then raise Step from 0 to 20. On f = x² + 3y², one step multiplies x by (1 − 2η) and y by (1 − 6η). Above η ≈ 0.17 the y factor turns negative and the path zigzags across the valley. At η = 0.15, y shrinks to a tenth each step while x keeps 70%, so most of the remaining distance is along x. This mismatch between steep and shallow directions is why adaptive optimizers such as Adam scale each partial derivative separately.

Step 6 of 6: Higher Dimensions

The same idea scales to millions of parameters

Number of dimensions2
2x14x2∇f = (2, 4)Each bar = ∂f/∂xᵢ at point (1,1,...,1)
Dimensions
2
|∇f|
4.47
Params
2
Max ∂f/∂xᵢ
4

In 2D, the gradient is a vector with 2 components, one partial derivative per variable. This is still small enough to picture.

Raise the Number of dimensions from 2 to 10 and watch the bars. The function is f = Σ (i+1)·xᵢ², evaluated where every coordinate equals 1, so the partial derivatives are 2, 4, 6, and so on. With 10 dimensions, |∇f| = 2√385 ≈ 39.2. Estimating each partial with a finite difference would need one extra evaluation of f per dimension. Backpropagation gets all of them for a small constant multiple of one evaluation, which is what makes billion-parameter training possible.

What is a partial derivative?

A partial derivative measures how a function of several variables changes when one input moves and all the others are held fixed. For f(x, y) it is written ∂f/∂x or ∂f/∂y. Collecting every partial derivative of a scalar function into one vector gives the gradient, ∇f, which points in the direction of steepest increase.

A neural network's loss is a scalar function of all its weights, so training needs one partial derivative per weight. Backpropagation computes that whole gradient, and optimizers such as SGD and Adam use it to update every parameter at once. The same idea underlies sensitivity analysis, feature attributions such as saliency maps, and physics simulations.

Key ideas

  • A partial derivative changes one input and holds all the others fixed.
  • The gradient ∇f is the vector of all partial derivatives of a scalar function.
  • The gradient points in the direction of steepest increase, and its length is that steepest slope.
  • The gradient is perpendicular to the contour lines of the function.
  • Zero gradient marks a critical point, which can be a minimum, maximum, or saddle.
  • Gradient descent steps opposite to the gradient, scaled by the learning rate.

Common questions

What is the difference between a partial derivative and a total derivative?

A partial derivative holds the other inputs fixed. A total derivative lets them change too, when they depend on the same underlying variable. If x and y both depend on t, then df/dt = (∂f/∂x)(dx/dt) + (∂f/∂y)(dy/dt), which adds up the effect through every path.

Does the order of partial derivatives matter?

For smooth functions it does not. Schwarz's theorem (also called Clairaut's) says ∂²f/∂x∂y = ∂²f/∂y∂x whenever both second partials are continuous. For f = x² + xy + y², both mixed partials equal 1.

What does a large partial derivative tell you about a feature?

It says the output is locally sensitive to that input: a small change there moves f a lot. Saliency maps for image models use this, showing ∂(class score)/∂(pixel) as a heatmap. The measure is local and depends on the input's scale, so a large value is not proof that a feature matters globally.

How do you get partial derivatives in PyTorch?

Mark the inputs with requires_grad=True, compute a scalar output, and call .backward() on it. Each input's .grad attribute then holds the partial derivative of the output with respect to that input. torch.autograd.grad returns the same values without storing them on the tensors.

Related explainers

References

  1. Deisenroth, Faisal & Ong, "Mathematics for Machine Learning", ch. 5 Vector Calculus
  2. Goodfellow, Bengio & Courville, "Deep Learning", ch. 4 Numerical Computation
  3. Simonyan, Vedaldi & Zisserman (2013), "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps"

Last updated