ENID
Lensa ML

How Derivatives Work

Step 1 of 6: It Starts With Slope

How steep is the curve at one exact point?

-1123xypoint (1.5, 2.3)slope = 3.0f(x) = x²
Drag to move the pointx = 1.50
Point
(1.5, 2.3)
Slope
3.00

The slope of the tangent line tells you how fast f(x) is changing at exactly that point. That slope is the derivative: the instantaneous rate of change.

Drag the point along f(x) = x² from x = −1 to x = 3 and compare the slope readout with the x value. The slope is always exactly twice x, a pattern the later steps prove. For a straight line, slope is rise over run and it is the same everywhere. A curve has a different slope at every point, so the question becomes which straight line best matches the curve near one point. That line is the tangent, and its slope is the derivative at that point.

Step 2 of 6: Two Points, One Line

Drag the second point closer — watch what happens to the slope

-0.50.51.52.53.5xyh = 1.50rise = 5.25a = 1a+hf(x) = x²
Drag the second point closer to a = 1h = 1.50
Secant slope
3.500
True derivative
2.000
Error
1.500

The secant line connects two points on the curve. Its slope = [f(a+h) - f(a)] / h. As you drag h smaller, the secant slope gets closer to the true derivative.

Drag the second point toward a = 1 and watch the error readout. For x², the secant slope is (f(1+h) − f(1)) / h = 2 + h, so the error equals h exactly: at h = 0.5 the slope is 2.5, at h = 0.05 it is 2.05. This difference quotient is also how numerical differentiation works in code. Libraries check hand-written gradients by comparing them with a centered version, (f(x+h) − f(x−h)) / 2h, whose error shrinks like h² instead of h.

Step 3 of 6: Shrinking Toward Zero

The secant line becomes a tangent as h vanishes

-0.50.51.52.53.54.5xyx = 1.5h = 2.000 → slope = 5.000

The secant cuts through the curve at two points. As h shrinks, the second point slides toward the first.

Press Watch h Shrink to Zero, or Reset to start again from h = 2. The animation multiplies h by 0.92 every frame, so it gets arbitrarily close to zero without reaching it. That is the point of a limit: the formula [f(x+h) − f(x)] / h is undefined at h = 0, yet its value as h approaches 0 is well defined. Here the fixed point is x = 1.5 and the slope settles at 3. Newton and Leibniz used this idea in the 1600s; the rigorous ε-δ definition came in the 1800s.

Step 4 of 6: The Algebra Behind It

Step through the limit computation for f(x) = x²

Start with the definition
f'(x) = limh→0 f(x+h) - f(x)h

The derivative is defined as this limit as h approaches 0.

Click Next Step to move through the six lines of the proof. The same pattern works for any power: expanding (x+h)ⁿ gives xⁿ + n·xⁿ⁻¹·h plus terms with h² or higher, so the derivative of xⁿ is n·xⁿ⁻¹. That is the power rule. In practice nobody takes limits by hand. Rules like this one, plus the product and chain rules, let software differentiate any formula built from known pieces, which is what automatic differentiation in PyTorch and JAX does.

Step 5 of 6: The Derivative Function

Every x has its own slope — that gives us a whole new function

-2.5-1.5-0.50.51.52.5xyf(x) = x²y = 1.00-2.5-1.5-0.50.51.52.5xyf'(x) = 2xslope = 2.00
Slide x to explore both functionsx = 1.00
f(1.0)
1.00
f'(1.0) = slope
2.00

The curve is climbing here, so f'(x) is positive. The steeper the curve, the higher the derivative.

Slide x from −2 to 2 and follow the point on both graphs. The lower curve, f'(x) = 2x, is a new function: each of its heights is a slope of the upper curve. Where f' crosses zero, f has a flat spot, which is how optimizers locate candidate minima. The derivative of f' is the second derivative, f''(x) = 2 here. A positive second derivative means the curve bends upward, which confirms that the flat point at x = 0 is a minimum and not a maximum.

Step 6 of 6: Why It Matters

Derivatives power everything from physics to machine learning

The definition of the derivative
f'(x) = limh→0 f(x+h) - f(x)h
Physics
Position → Velocity → Acceleration. Each is the derivative of the one before it.
Economics
Marginal cost is the derivative of total cost. It gives the cost of making one more unit.
Machine Learning
Gradient descent uses derivatives to find the minimum of a loss function. This is how neural networks learn.
Engineering
Rate of heat flow, stress analysis, signal processing: all built on derivatives.

The derivative turns the question "how fast is this changing?" into a precise, computable answer. Rocket trajectories and neural network training both depend on it.

Slope of tangent
Instantaneous rate
lim h→0
f'(x) = 2x

Click any card to read it; the cards also cycle on their own. In machine learning the function being differentiated is the loss, and there is one derivative per parameter, often billions of them. Gradient descent updates each parameter as w ← w − η·∂L/∂w, where η is the learning rate. The derivative supplies the direction and the learning rate sets the step size. In physics the chain is position, velocity, acceleration, each the derivative of the one before.

What is a derivative?

A derivative measures how fast a function's output changes as its input changes, at one specific point. Geometrically it is the slope of the tangent line to the curve there. Formally it is the limit f'(x) = lim h→0 [f(x+h) − f(x)] / h, the slope of a line through two points as the points merge into one.

Machine learning runs on derivatives. Training a model means adjusting its parameters to reduce a loss, and the derivative of the loss with respect to each parameter says which direction lowers it and by roughly how much. Gradient descent, backpropagation, learning-rate schedules, and second-order methods such as Newton's method are all built from this one idea.

Key ideas

  • The derivative at a point is the slope of the tangent line there.
  • It is defined as the limit of secant slopes, [f(x+h) − f(x)] / h, as h approaches 0.
  • The derivative of x² is 2x, and the power rule gives n·xⁿ⁻¹ for xⁿ.
  • A positive derivative means the function is increasing; a negative one means it is decreasing.
  • Points where the derivative is zero are candidates for minima and maxima.
  • Gradient descent uses derivatives of the loss to decide how to change each parameter.

Common questions

What is the difference between a derivative and a differential?

The derivative f'(x) is a rate: output change per unit of input change. The differential dy = f'(x)·dx is the approximate change in output for a small input change dx. The notation dy/dx, due to Leibniz, treats the derivative as a ratio of these small changes.

Can a function fail to have a derivative?

Yes. The function |x| has a sharp corner at 0, where the slope from the left is −1 and from the right is +1, so no single tangent exists. ReLU has the same kink at 0. Frameworks pick a value there, usually 0, and training works fine because inputs land exactly on 0 so rarely.

What are the derivatives of eˣ and ln x?

The derivative of eˣ is eˣ, which makes it the only function (up to a constant multiple) equal to its own derivative. The derivative of ln x is 1/x. Both appear constantly in ML because softmax, sigmoid, and cross-entropy loss are built from exponentials and logarithms.

Why not just compute derivatives numerically with a small h?

Finite differences need one or two extra function evaluations per input, which is far too slow for a model with millions of parameters. They also lose accuracy: too large an h gives truncation error, too small an h gives floating-point cancellation. Automatic differentiation computes exact derivatives for all parameters at roughly the cost of one extra pass.

Related explainers

References

  1. Strang, "Calculus" (open textbook), MIT OpenCourseWare
  2. Deisenroth, Faisal & Ong, "Mathematics for Machine Learning", ch. 5 Vector Calculus
  3. Goodfellow, Bengio & Courville, "Deep Learning", ch. 4 Numerical Computation

Last updated