1Optimizers 2Learning Rate Scheduling 3Weight Initialization 4Batch Normalization 5Vanishing & Exploding Gradients 6Overfitting & Underfitting 7Regularization

Lensa ML

How Weight Initialization Works

Step 1 of 5: Constant Initialization

Why zeros and ones both fail as starting weights

Constant Weight: 0.50

Weight

0.50

H1 Value

0.291

H2 Value

0.525

Unique Out

All weights identical → every neuron in a layer is a clone. Their gradients are identical too, so they stay clones after every update. Random values are needed to break this symmetry.

Drag the constant weight from 0 to 2 and tap any node to see its calculation. The failure has a name: symmetry. If two neurons in a layer start with the same incoming and outgoing weights, they compute the same output and receive the same gradient. After every update they are still identical, so a layer of width n behaves like a single neuron. Biases are a different case. Starting all biases at zero is standard and harmless, because the random weights already make the neurons differ.

Step 2 of 5: Random Initialization

Random values beat constants, but the scale matters

Uniform Bound [-0.50, 0.50]

Bound

0.50

H1 Std

0.385

H2 Std

0.238

Grad @ Input

0.322

Symmetry is broken (each neuron in the diagram has its own value), but the bound is still too small. Each layer passes on a narrower spread than it received, so the histograms shrink layer by layer and the gradient shrinks with them. With 3–4 inputs per neuron, a bound near 1 keeps the spread steady.

Drag the uniform bound from 0.01 to 3 and watch the layer histograms and the gradient card. The scale problem has a simple cause. A neuron sums n weighted inputs. If the inputs and weights are independent with zero mean, the variance of that sum is n·Var(w)·Var(x). With a fixed bound such as ±0.5, the output variance grows in proportion to the layer width. A setting that works for 10 inputs saturates every tanh unit when a layer has 1,000 inputs.

Step 3 of 5: The 1/√n Rule

A simple recipe that adapts initialization to layer width

Base Width: 32Network Shape

Weight Distribution

32 → 32 → 32 → 32 → 32 → 32(σ = 1/√n = 0.177, 0.177, 0.177, 0.177, 0.177)

Shape

Uniform

Widths

32→32→32→32→32

Naive L5 Std

0.643

Recipe L5 Std

0.320

The recipe sets σ = 1/√n, so a sum of n weighted inputs has about the variance of one input: Var(z) ≈ n·σ²·Var(x) = Var(x). Its L5 std (0.320) comes out about the same at any width. Naive gives 0.643 at this width, but that number changes as you move the width slider. Tanh still squeezes values a little in every layer, which is why the recipe’s std drifts down gently.

Drag the base width from 8 to 64 and switch between the Uniform, Taper, and Expand shapes. Setting Var(w) = 1/n cancels the n in n·Var(w)·Var(x), so each layer passes on roughly the variance it received. This 1/√n scaling goes back to LeCun et al.'s "Efficient BackProp" (1998) and is often called LeCun initialization. A uniform distribution on [−a, a] has variance a²/3. For a uniform draw to match a target variance, the bound must therefore be √3 times the target standard deviation.

Step 4 of 5: Xavier, He & Activations

Named methods tuned for specific activation functions

Base Width: 16Network Shape

Activation Function

Weight Distribution

16 → 16 → 16 → 16 → 16 → 16

Shape

Uniform

Widths

16→16→16→16→16

Xavier Last Std

0.329

He Last Std

0.576

Both Xavier and He are variations of the 1/√n rule. Each comes in two flavors: Normal (sample from a Gaussian) and Uniform (sample from a flat distribution with the same variance). Currently using Normal: Xavier ~ N(0, 2/(n_in+n_out)), He ~ N(0, 2/n_in), where the second number is the variance σ². Xavier (Glorot, 2010) averages fan-in and fan-out for symmetric activations (tanh, sigmoid). He (Kaiming, 2015) uses only fan-in with a ×2 factor for ReLU. With tanh: Xavier last std = 0.329, He last std = 0.576.

Switch the activation between tanh and relu and compare the two last-layer standard deviations. Glorot and Bengio (2010) balanced two goals: keep activation variance stable on the forward pass (Var(w) = 1/n_in) and gradient variance stable on the backward pass (Var(w) = 1/n_out). Their compromise is 2/(n_in + n_out). He et al. (2015) noted that ReLU zeros about half its inputs, which halves the variance at each layer. Doubling the variance to 2/n_in compensates, and it let them train a 30-layer ReLU network from scratch.

Step 5 of 5: Training Comparison

Watch how different initializations affect actual learning

Network Width: 16

Epoch 0

ZerosConst 0.1Random1/√n

Epoch

1/√n Loss

1.476

Best

Const 0.1

Weight Std

0.332

1/√n recipe → loss = 1.476 at this epoch, while Const 0.1 is lowest right now (0.354). The weight distribution stays centered near zero with a controlled spread, because the scale was set from each layer’s width.

Choose each method and press Play, or drag the epoch slider, at different network widths. The network is 1 → width → width → 1 with ReLU, fit to 16 points of y = sin(x) for 80 epochs. The Const 0.1 option starts every weight at 0.1. Starting at 1 instead would push the first-epoch loss above 100,000 at width 16. In this small network the 1/√n recipe reaches the lowest loss at every width, and naive ±0.3 random weights catch up only near width 30. Batch normalization and residual connections reduce, but do not remove, a deep network's sensitivity to these choices.

What is weight initialization in neural networks?

Weight initialization is the choice of starting values for a neural network's weights before training begins. Good schemes draw small random values with a variance scaled to each layer's size, so that activations and gradients keep a stable magnitude from layer to layer. Bad choices make every neuron identical, or make signals shrink to zero or grow without bound as they pass through the network.

Every framework applies an initializer by default, and the choice matters most for deep networks. Xavier (Glorot) initialization is the standard for tanh and sigmoid layers, and He (Kaiming) initialization for ReLU layers. Transformers commonly use small normal initializations, with residual projections sometimes scaled down further by depth, as in GPT-2.

Key ideas

Identical starting weights make neurons identical, and gradient descent cannot separate them.
Random weights break that symmetry, but their scale must match the layer size.
The variance of a neuron's weighted sum grows with the number of inputs it receives.
Scaling weight variance by 1/n_in keeps activations at a stable size across layers.
Xavier initialization suits tanh and sigmoid; He initialization suits ReLU.
Biases are usually initialized to zero.

Common questions

What is the difference between Xavier and He initialization?

Xavier sets the weight variance to 2/(n_in + n_out), which suits activations that are roughly linear near zero, such as tanh. He sets it to 2/n_in, twice as large as LeCun's 1/n_in, to make up for ReLU discarding the negative half of its inputs. Using Xavier with deep ReLU networks makes activations shrink layer by layer.

What initialization does PyTorch use by default?

PyTorch's nn.Linear and nn.Conv layers use a Kaiming-uniform scheme with a=√5, which works out to a uniform distribution on ±1/√n_in for the weights. Biases are drawn uniformly from the same range. Many model codebases override this with explicit Xavier, He, or small normal initializations.

Should I use a normal or uniform distribution?

Either works if the variance is right. Normal and uniform initializations with matched variance give very similar results. Some codebases use a truncated normal, which redraws values beyond two standard deviations, to avoid rare large weights.

Does weight initialization matter if I use batch normalization?

Less, because batch normalization rescales each layer's activations regardless of the weight scale. Initialization still affects the earliest training steps and any layers without normalization. Residual networks also benefit from special tricks, such as starting the last layer of each block at zero so the block begins as the identity.

Related explainers

Activation Functions — Why networks need non-linearity to learn
Vanishing & Exploding Gradients — Why deep networks struggle and how to fix it
Batch Normalization — Why normalizing activations speeds up deep network training
Backpropagation — How networks learn from their mistakes
Neural Networks — Neurons, layers, and how networks learn
Skip Connections & ResNets — Residual learning and why deeper networks work

References

Last updated October 25, 2026