How Weight Initialization Works

All weights identical → every neuron in a layer is a clone. Gradients are identical too — the network can never learn different features. Random values are needed to break this symmetry.

Symmetry broken! Each neuron computes a different value (see the network above). The histograms show healthy spread across layers. But does this scale work for all layer widths? We need a principled rule.

The recipe y = 1/√n scales initialization to the layer width. Uniform sampling keeps activations stable (L5 std = 0.047) while naive uniform drifts (L5 std = 0.643).

Both Xavier and He are variations of the 1/√n rule. Each comes in two flavors: Normal (sample from a Gaussian) and Uniform (sample from a flat distribution with the same variance). Currently using Normal: Xavier ~ N(0, √(2/(n_in+n_out))), He ~ N(0, √(2/n_in)). Xavier (Glorot, 2010) averages fan-in and fan-out for symmetric activations (tanh, sigmoid). He (Kaiming, 2015) uses only fan-in with a ×2 factor for ReLU. With tanh: Xavier last std = 0.329, He last std = 0.576.

1/√n recipe → fastest convergence (loss = 0.378). Weight distribution stays centered near zero with controlled spread. This is what the W&B article demonstrates: principled initialization beats everything.