The untrained network sees a cat photo's three features and outputs Cat 0.571, Dog 0.671: it calls the cat a dog. The loss adds up the squared misses, (0.571 − 1)² + (0.671 − 0)² = 0.635. The amber line is the loss's slope for the Cat output, ∂L/∂o = 2(o − 1) = -0.858: negative, so raising the Cat output would lower the loss. That slope is where backpropagation starts. The question is how each of the 36 weights should move to achieve it.
Backpropagation answers a credit-assignment question: this network has 36 weights, so which of them caused the wrong answer, and by how much? Drag the hypothetical Cat output and watch the slope of the loss change sign as it passes the target. That slope, ∂L/∂o, is the only thing backprop needs from the loss. This explainer uses squared error, whose slope is 2 × (output − target). Classifiers usually train with cross-entropy instead, because squared error flattens out when a sigmoid output is confidently wrong, and learning slows down there.
Step 2 of 6: One Weight, One Chain
Multiply three local slopes, then check by nudging
Nudge w by ε0.010
∂L/∂w
-0.1376
Predicted change ε × ∂L/∂w
-0.00138
Actual change (re-run the network)
-0.00137
Take the highlighted weight, from the first Hidden-2 neuron to Cat. A change in it passes through three local steps, and the chain rule multiplies their slopes: the loss's slope 2(o − 1) = -0.858, the sigmoid's slope o(1 − o) = 0.245, and the input this weight carries, h = 0.655. The product, -0.1376, is the gradient. Check it: nudge the weight by ε = 0.010 and the gradient predicts the loss changes by -0.00138; re-running the network gives -0.00137. For small nudges they match.
Drag ε and compare the two change readouts. The chain-rule gradient predicts the loss will change by ε × ∂L/∂w; the other card actually re-runs the network with the nudged weight. They agree for small nudges and drift apart for big ones, because a gradient is the slope at one point. This comparison, called a gradient check, is how people test hand-written backprop code. The last factor in the chain, ∂z/∂w, is just the value the weight carries, so a weight attached to a near-zero input gets almost no blame.
This weight sits one layer deeper, from Hidden-1 to Hidden-2. Changing it changes the second Hidden-2 neuron, which feeds both outputs, so it affects the loss along two paths. The chain rule multiplies the slopes along each path and adds the paths: 0.01060 via Cat plus 0.02079 via Dog = 0.03138, matching the numerical check 0.03138. In a big network one weight has millions of paths. Backprop never lists them: it notices that every path through a neuron shares the part after that neuron, and computes that shared part once.
Switch between the two paths. This weight feeds one Hidden-2 neuron, and that neuron feeds both outputs, so the weight reaches the loss twice. The chain rule handles that by multiplying the slopes along each path and adding the paths together, and the sum matches the numerical check. A weight in the first layer of a large network has a huge number of paths to the loss. Backprop never lists them, because every path through a neuron shares the part after that neuron, and that shared part is computed once.
Step 4 of 6: Gradients Flow Backward
Each δ is built from the δs after it
Each circle shows its δ = ∂L/∂z. Click a hidden or output neuron.
A hidden neuron's δ is its own sigmoid slope, 0.177, times the weighted sum of the δs it feeds into (the amber edges). That's the whole algorithm: compute the output δs, then sweep backwards layer by layer reusing them, and every weight's gradient is just δ of the neuron it enters times the value it carries. Notice the sizes: the average |δ| shrinks 47× from the output to Hidden 1, because each layer multiplies by a sigmoid slope of at most 0.25. In deep sigmoid networks that is the vanishing-gradient problem.
Click any hidden or output neuron to see its δ (delta, the gradient of the loss with respect to the neuron's weighted sum) built from the deltas after it. An output's δ comes straight from the loss. A hidden neuron's δ is its own sigmoid slope times the weighted sum of the deltas it feeds. The bars show the average size per layer: it shrinks by roughly 47× between the output and Hidden 1, because each layer multiplies by a sigmoid slope of at most 0.25. That is the vanishing-gradient problem in miniature. Frameworks such as PyTorch do this bookkeeping automatically, a technique called reverse-mode automatic differentiation.
Step 5 of 6: Nudging the Weights
One step helps this photo, but what about the next?
Learning rate2.0
Cat photo loss
0.635 → 0.300
Cat photo: Cat · Dog
0.71 · 0.46
Dog photo loss
0.437 → 0.797
Dog photo is called
Cat
Every weight moves at once by w ← w − rate × ∂L/∂w, each by its own gradient. At rate 2.0 the cat photo's loss goes from 0.635 to 0.300. The cat photo is now called Cat, but the dog photo, which this step never saw, is now called Cat too, with loss 0.797. The step didn't teach the network what a cat looks like; the cheapest way to fix one cat photo was to lean the whole network towards Cat. Here the danger isn't overshooting this one photo: its loss only falls as the rate grows. The danger is that a gradient from one example only knows about that example. Real training averages the gradients over a batch of cat and dog photos and takes small steps, so each update helps on average.
Raise the learning rate and watch both curves. The cat photo's loss only falls as the rate grows, so overshooting isn't the problem here. The dog photo, which the step never saw, starts out correct and is soon called Cat as well. One example's gradient says how to fix that example, and on this untrained network the cheapest fix is to lean towards Cat for every input. Real training averages gradients over a mini-batch of varied examples and takes many small steps, often with rates below 0.001 and optimizers such as Adam that scale each weight's step.
Step 6 of 6: All Gradients in One Pass
Why backprop beats nudging every weight
Mean |∂L/∂w|, last layer
0.147
Mean |∂L/∂w|, first layer
0.0044
Network runs: backprop
1 forward + 1 backward
Network runs: nudge each weight
37 forward
One backward sweep produced all 36 gradients. Getting them by nudging each weight and re-running the network, as in step 2, would take 37 forward passes here, and billions for a large model, while backprop costs about one extra pass whatever the size. That's why it trains every modern network. The bars also show which weights matter right now: the last layer's gradients average 0.147, the first layer's only 0.0044, so the early layers will learn slowest. Gradient descent then repeats forward pass, backward pass, update until the loss stops falling.
Filter the bars by layer and compare the averages. The last layer's gradients are about 30× larger than the first layer's, so early layers would learn slowest if training started now. Every bar came from the same single backward sweep. Estimating them by nudging each weight, as in step 2, would take 37 forward passes here, and one per weight for a model with billions of weights. Backprop costs about the same as one extra forward pass whatever the size. That gap is why every modern network trains with it.
What is backpropagation?
Backpropagation is the algorithm that computes, for every weight in a neural network, how much the loss would change if that weight changed slightly. That quantity is the weight's gradient. Backprop applies the chain rule from the output layer back toward the input and reuses intermediate results, so one backward pass yields the gradient of every weight. An optimizer such as gradient descent then uses those gradients to update the weights.
The method was popularized for neural networks by Rumelhart, Hinton, and Williams in 1986, building on earlier work on reverse-mode differentiation. Every modern deep learning framework, including PyTorch and JAX, runs a generalized version of it automatically.
Key ideas
Backprop computes the gradient of the loss with respect to every weight.
It applies the chain rule layer by layer, from the output back to the input.
One backward pass, at roughly twice the cost of a forward pass, yields every gradient at once.
A weight's gradient depends on the error signal arriving at its neuron and the input it carries.
Gradients can shrink as they travel back through many layers (vanishing gradients).
Backprop finds the gradients; the optimizer decides how to use them.
Common questions
Is backpropagation the same as gradient descent?
No. Backpropagation computes the gradients. Gradient descent, or another optimizer such as Adam, uses those gradients to change the weights. You can pair backprop with many different optimizers.
Why is it called backpropagation?
The error signal starts at the output, where the loss is measured, and is propagated backward through the network one layer at a time. The full name is backward propagation of errors.
How does ReLU change backpropagation?
ReLU's derivative is 1 for positive inputs and 0 for negative ones. Gradients pass through active neurons without shrinking, which helps deep networks train, but a neuron that is negative for every input stops receiving any gradient. That failure is called a dead ReLU.
Why does backpropagation use so much memory?
The backward pass needs the activations computed during the forward pass, so they are kept in memory until the gradients are done. Memory grows with network depth and batch size. Gradient checkpointing reduces it by storing only some activations and recomputing the rest during the backward pass.