1Optimizers 2Learning Rate Scheduling 3Weight Initialization 4Batch Normalization 5Vanishing & Exploding Gradients 6Overfitting & Underfitting 7Regularization

Lensa ML

How Batch Normalization Works

Step 1 of 6: The Problem

Why shifting data kills learning

Data flowing through layers:

Where output data lands on the activation function:

How much data shifts (drag me!)0.0

Data drift

0.0

Avg gradient

0.248

Gradient health

Strong

Data flows from Input to Output. The dots land on the steep middle of the sigmoid (green dots), where gradients are strong and the network learns quickly.

Drag the shift slider from 0 to 3 and watch the gradient readouts. Each layer's input distribution depends on every weight before it, so it changes whenever those weights update. Ioffe and Szegedy called this internal covariate shift and proposed BN to remove it. Later work questioned that explanation. Santurkar et al. (2018) found that BN still helps when noise is deliberately injected after it, and argued that its real benefit is a smoother loss surface, which makes gradients more predictable and larger steps safe.

Step 2 of 6: Where BN Lives

Placed between linear layer and activation

The BN pipeline: x → Wx+b → BN → activation → output (x = 1.5 is one sample in a batch of 4: 0.5, 1.0, 1.5, 2.5)

Where the value lands on the sigmoid:

Weight scale1.0

Pre-BN

1.8

Post-BN

1.8

Gradient

0.122

Without BN, small weights keep values in the sigmoid's useful range. Training often pushes weights larger, though.

Toggle Batch Norm and drag the weight scale from 0.5 to 4. The original paper placed BN after the linear transform and before the nonlinearity, as shown here. When BN follows a linear layer, that layer's bias becomes redundant, since BN subtracts the mean and then adds its own β. Frameworks therefore build such layers with bias turned off. Some architectures put BN after the activation instead, and in practice the difference is small. In a CNN, BN computes one mean and variance per channel, over the batch and all spatial positions.

Step 3 of 6: BN in Action

Watch the calculation inside a network

A batch of 4 inputs flowing through one layer:

Before and after BN (same values on a number line):

Weight (W)1.5

Bias (b)0.5

Batch mean

2.30

Batch std

1.18

Norm range

-1.3…1.4

Four inputs go through Wx+b, producing scattered pre-activation values (top line). BN computes the batch mean and std, then normalizes each value as (x − mean) / std, centering them at zero (bottom line).

Drag the weight W and bias b and compare the batch mean and std cards. The full formula is x̂ = (x − μ_B)/√(σ²_B + ε), then y = γ·x̂ + β, where μ_B and σ²_B come from the current mini-batch and ε (1e-5 here) prevents division by zero. With four values the batch statistics are noisy, and each example's output depends on the other three. That noise acts as a regularizer, but it also means BN behaves poorly with tiny batches, roughly below 8 to 16 examples per device.

Step 4 of 6: Scale & Shift

Learnable γ and β restore power

γ (scale)1.00

β (shift)0.00

1.00

0.00

Output range

-1.5…1.5

γ=1, β=0: the output equals the normalized values. This is the starting point before the network learns the best scale and shift.

Drag γ and β, then press Restore original. The example batch is 2, 5, −1, 8, 3, with mean 3.4 and standard deviation about 3.0, so γ = 3.0 and β = 3.4 undo the normalization exactly. γ and β are learned per feature, like any other weight, and start at 1 and 0. A common trick in residual networks initializes γ to 0 in the last BN of each block. Each block then starts as the identity function, which makes very deep networks easier to train at the start.

Step 5 of 6: Train vs Inference

Batch stats vs running averages

Batch means (dots) vs running average (line) over training:

Batch meanRunning avg

Training

uses batch stats

mean=2.1

normalized=0.27

Inference

uses running avg

mean=0.2

normalized=2.18

Batch index0

Momentum0.10

Batch mean

2.1

Running mean

0.21

Difference

1.91

During training, BN uses the current batch's mean (orange dot). It also keeps a running average (blue line) that tracks the batch means over time. Training and inference give different outputs for the same input value.

Step through the batch index and drag the momentum slider. The running mean updates as μ_run = (1 − m)·μ_run + m·μ_B, so momentum m sets how fast it forgets old batches. PyTorch's default is 0.1. Keras uses the opposite convention, with a default of 0.99 on the old value. Calling model.eval() in PyTorch switches BN to the running statistics. Forgetting that call is a common bug: predictions then change with batch composition, and single-example inference gives poor results.

Step 6 of 6: The Payoff

Faster training, higher learning rates

With BNWithout BN

Epoch0

Learning rate0.010

0.010

Loss (BN)

2.65

Loss (no BN)

2.90

At this conservative learning rate, both models learn slowly. Press Play to watch them train. BN has a slight edge even here. These curves are an illustrative sketch of typical behaviour, not a real training run.

Press Play, or step with −1 and +1, then raise the learning rate past 0.04 and replay. Ioffe and Szegedy reported that an Inception network with BN matched the original's ImageNet accuracy in 14 times fewer training steps, and that BN made much higher learning rates usable. BN also made it practical to train networks with saturating activations like sigmoid. The curves here are a stylized simulation of that effect, not a trained model, but the pattern matches what practitioners see.

What is batch normalization?

Batch normalization (BN) is a layer that normalizes each feature to zero mean and unit variance using statistics from the current mini-batch, then applies a learned scale γ and shift β. It is usually placed between a linear or convolutional layer and its activation. At inference time it uses running averages of the mean and variance collected during training.

Ioffe and Szegedy introduced batch normalization in 2015, and it quickly became standard in convolutional networks such as ResNet. It allows higher learning rates, makes training less sensitive to initialization, and adds a mild regularizing effect. Transformers mostly use layer normalization instead, which normalizes across the values within each example and does not depend on the batch.

Key ideas

Batch normalization rescales each feature to zero mean and unit variance within a mini-batch.
Learned parameters γ and β let the network restore any scale and shift it needs.
Training uses batch statistics; inference uses running averages collected during training.
BN permits higher learning rates and reduces sensitivity to weight initialization.
Small batches make BN's statistics noisy, which is why transformers use layer normalization.
Why BN works is still debated; smoothing the loss surface is a leading explanation.

Short takes on this topic

Batch normalization in plain words

Common questions

What is the difference between batch normalization and layer normalization?

Batch normalization computes the mean and variance of each feature across the examples in a batch. Layer normalization computes them across all channels or hidden units of a single example. Layer norm does not depend on batch size and behaves the same in training and inference, which suits transformers and recurrent networks.

Should batch normalization go before or after the activation?

The original paper put it before the activation, right after the linear or convolutional layer, and that is the most common choice. Placing it after the activation also works and sometimes performs slightly better. The choice rarely changes results much.

Can I use batch normalization and dropout together?

You can, but they can interfere: dropout changes the variance of activations between training and inference, which shifts the statistics BN relies on. Networks that use BN often need little or no dropout. If both are used, dropout is usually placed after the last BN layer.

Why does my model perform worse in evaluation mode with batch norm?

In evaluation mode BN uses running averages instead of batch statistics. If those averages are poor, for example after very few training steps, after fine-tuning on different data, or with a momentum set too low, outputs shift. Recomputing the running statistics on training data usually fixes it.

Related explainers

Feature Scaling — Why and how to normalize features for faster convergence
Vanishing & Exploding Gradients — Why deep networks struggle and how to fix it
Weight Initialization — Xavier, He, and why random weights matter
Activation Functions — Why networks need non-linearity to learn
Convolutional Neural Networks — How CNNs see and understand images
Skip Connections & ResNets — Residual learning and why deeper networks work

References

Last updated October 25, 2026