How Batch Normalization Works
The Problem
Why shifting data kills learning
Data flows from Input to Output. The dots land on the steep middle part of the sigmoid curve (green dots). Gradients are strong here, so the network learns quickly!
Step 1 of 6: The Problem
Why shifting data kills learning
Data flows from Input to Output. The dots land on the steep middle part of the sigmoid curve (green dots). Gradients are strong here, so the network learns quickly!
Step 2 of 6: Where BN Lives
Placed between linear layer and activation
Without BN, small weights keep values in the sigmoid's useful range. But real networks need large weights to learn complex patterns.
Step 3 of 6: BN in Action
Watch the calculation inside a network
Four inputs go through Wx+b, producing scattered pre-activation values (top line). BN looks at the whole batch, computes the mean and std, then normalizes each value: (x - mean) / std. Result: always centered at zero (bottom line).
Step 4 of 6: Scale & Shift
Learnable γ and β restore power
γ=1, β=0: the output equals the normalized values. This is the starting point before the network learns the best scale and shift.
Step 5 of 6: Train vs Inference
Batch stats vs running averages
During training, BN uses the current batch's mean (orange dot). It also keeps a running average (blue line) that smoothly tracks over time. Notice how training and inference produce different outputs for the same input value.
Step 6 of 6: The Payoff
Faster training, higher learning rates
At this conservative learning rate, both models learn slowly. Press Play to watch them train. BN has a slight edge even here.