ENID
Lensa ML

Activation Functions

Step 1 of 6: Why Not Just Linear?

Bends build shapes; without them, layers collapse

-1.501.5-303x
- - target sin(1.2x) · ━ network output · thin = each unit
Hidden units4
Fit error (RMSE)
0.210
Bends in the output
4
Whole network equals
a bent curve

Each ReLU unit outputs 0 until its input passes a threshold, then rises in a straight line: one bend (the white dots). Weighted and added up, 4 bends trace the wave with error 0.210; here the output weights are the best possible (least squares). More units, more bends, a closer fit. This is the universal-approximation idea in its simplest form. Switch the activation off to watch it collapse.

Change the number of hidden units and switch the activation off. A layer computes W·x + b, and composing two of them, W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂), gives another single layer, so without a non-linearity depth adds nothing. With ReLU each unit adds one bend, and the universal approximation theorem says a wide enough layer of such units can approximate any continuous function on a bounded range. Here the output weights are fitted by least squares, the best possible for the given bends.

Step 2 of 6: The Sigmoid

A slope of at most 0.25, multiplied every layer

01-606x
━ σ(x) · - - σ′(x)
x1.5
gradient after 5 layers (log scale)sigmoid at this x7.4e-5sigmoid, best case9.8e-4ReLU, active unit1.000
Layers deep5
σ(x)
0.818
Slope σ′(x)
0.149
σ′ multiplied 5 times
7.4e-5

Sigmoid squashes any number into (0, 1), handy for a probability. But its slope σ′(x) = σ(x)(1 − σ(x)) is at most 0.25, at x = 0, and here it is 0.149. Backpropagation multiplies one such slope per layer, so through 5 sigmoid layers the gradient shrinks by at least 0.25^5 = 9.8e-4. Early layers barely learn: the vanishing gradient. An active ReLU has slope exactly 1, so it passes the gradient through untouched.

Move x and the depth. The sigmoid σ(x) = 1/(1 + e⁻ˣ) has derivative σ(x)(1 − σ(x)), at most 0.25 at x = 0. Backpropagation multiplies one such factor per layer, so a 10-layer sigmoid network shrinks gradients by at least 0.25¹⁰ ≈ 10⁻⁶, the vanishing-gradient problem that held back deep networks until ReLU, better initialization, and residual connections. Sigmoid is still the right choice for a binary output and for the gates inside LSTMs.

Step 3 of 6: ReLU and Dead Neurons

Zero for negatives, and when that becomes a problem

-104-606pre-activation z = w·x + b
━ ReLU(z) · - - slope · bars: this neuron's z over 300 inputs
Bias b0.0
Inputs where it's active
52%
Inputs that send gradient back
156 / 300
Neuron status
alive

ReLU(z) = max(0, z): free to compute, slope exactly 1 when z > 0 and 0 when z < 0. A negative input only means this input sends no gradient through this neuron; the red bars are 48% of inputs doing that right now, which is normal. The neuron still learns from the other 52%. It only dies if z is negative for every input. Drag the bias down to see that happen.

Drag the bias down until the neuron dies. ReLU(x) = max(0, x) is cheap and has slope 1 for every positive input, so active units pass gradients through unchanged. Its derivative at exactly 0 is undefined, and frameworks simply use 0 there. A unit only dies when its pre-activation is negative for every input, often after a large update or a high learning rate; in practice some networks lose a noticeable share of units this way, which is what Leaky ReLU and careful initialization guard against.

Step 4 of 6: Tanh — Centered Sigmoid

Zero-centred outputs and a steeper slope

-101-404x
━ tanh(x) · ··· 2σ(2x) − 1 · ━ σ(x)
x1.0
tanh(x)
0.7616
2σ(2x) − 1
0.7616
Mean output, σ vs tanh
0.50 vs -0.01
Max slope, σ vs tanh
0.25 vs 1

tanh is a sigmoid stretched and shifted: tanh(x) = 2σ(2x) − 1, and the two readouts agree to every digit (0.7616). The change matters in two ways. Its outputs are centred on zero: over 300 typical inputs the mean is -0.01 for tanh but 0.50 for sigmoid, and all-positive outputs make the next layer's weight gradients share one sign, which forces a zig-zag path. And its slope peaks at 1, not 0.25, so gradients shrink 4× less per layer. It still saturates beyond |x| ≈ 2, which is why ReLU replaced both in hidden layers.

Slide x and compare the readouts. Tanh is a rescaled sigmoid, tanh(x) = 2σ(2x) − 1, with outputs in (−1, 1) centred on zero and a peak slope of 1 instead of 0.25. LeCun et al. (1998) recommended it over the sigmoid for hidden layers for exactly these reasons. It is still common in recurrent networks, where it squashes the LSTM cell state, but like the sigmoid it saturates for large |x|.

Step 5 of 6: The Activation Zoo

Leaky ReLU, ELU, GELU, Swish, and their slopes

-1.203-403x
━ GELU · - - its slope · ━ the others
x-1.0
f(x)
-0.159
Slope f′(x)
-0.083
Slope at x = −3
-0.012
Lowest output
-0.170

GELU is used in BERT, GPT-2 and most Transformers. Every member of the family behaves like ReLU for large positive x (slope near 1). They differ below zero: at x = −3 the slope is -0.012, so negative inputs still send a little gradient, and the curve is smooth through 0, which some optimizers like. It even dips slightly below zero (lowest -0.170), so it isn't monotonic. In practice the differences are small; ReLU or GELU is a sensible default.

Pick an activation and move x. Leaky ReLU keeps a slope of 0.01 to 0.1 below zero so units cannot die. ELU = eˣ − 1 for negatives is smooth when α = 1. GELU, x·Φ(x), weights the input by the probability that a standard normal is below it, and Swish or SiLU, x·σ(x), is a close cousin. GELU and Swish both dip slightly below zero, which some argue helps optimization, and a gated variant, SwiGLU, is used in LLaMA's feed-forward layers.

Step 6 of 6: Activations in Action

Train a real network on a ring

2 → 3 → 1 network, trained here with Adam
Training accuracy
100%
Final loss
0.002
Parameters
13

With 3 ReLU units the boundary is a polygon made of 3 straight folds, enclosing the ring at 100%. More units give more sides and a rounder shape. This network was actually trained in your browser: 13 parameters, 1500 steps of Adam on binary cross-entropy.

Switch the activation and the width, and compare the accuracy. These networks really train in your browser: 2 inputs, k hidden units, 1 sigmoid output, 1500 steps of Adam on 120 points. With no activation the boundary is always a line, near 57% on a ring. Two ReLU units can only make a wedge, three can enclose the ring, and more units round the polygon off. Real networks stack many such layers, so each layer can build features out of the previous layer's bends.

What is an activation function in a neural network?

An activation function is a nonlinear function applied to each neuron's weighted sum before the result passes to the next layer. Without it, a neural network of any depth computes only a linear function of its input. Common choices are sigmoid, tanh, ReLU, and smooth ReLU variants such as GELU and Swish.

The choice of activation affects what a network can represent and how well gradients flow during training. ReLU made deep convolutional networks practical around 2012. GELU is the default in BERT and GPT-style transformers, and a Swish-based gated unit (SwiGLU) is used in the feed-forward layers of LLaMA. Sigmoid survives at output layers for binary classification and in the gates of LSTMs.

Key ideas

  • Without a nonlinear activation, stacked layers collapse into a single linear layer.
  • Sigmoid and tanh saturate for large inputs, where their gradients approach zero.
  • Tanh is a shifted and rescaled sigmoid with zero-centered outputs.
  • ReLU is cheap, does not saturate for positive inputs, and is the default for many hidden layers.
  • Leaky ReLU, ELU, Swish, and GELU keep a nonzero response for negative inputs.
  • Output layers use task-specific activations: sigmoid for binary, softmax for multi-class, none for regression.

Common questions

Which activation function should I use?

For hidden layers, start with ReLU in convolutional and plain feed-forward networks, and GELU or SwiGLU in transformers. For the output layer, the choice follows the task: sigmoid for one probability, softmax for several mutually exclusive classes, and a linear output for regression.

What is the difference between an activation function and softmax?

Most activation functions act on each neuron independently. Softmax acts on a whole vector: it exponentiates every entry and divides by the sum, so the outputs are positive and add to 1. It is used at the output of classifiers to turn scores into class probabilities.

Does the activation function need to be differentiable?

It needs a usable gradient almost everywhere, because training relies on backpropagation. ReLU is not differentiable at 0, but that single point has no practical effect and frameworks assign it a derivative of 0. A step function, by contrast, has zero gradient everywhere it is defined, so gradient descent cannot train through it.

Why is GELU used in transformers?

GELU weights each input by the probability that a standard normal variable falls below it, giving a smooth curve instead of ReLU's sharp corner. Its authors reported small gains over ReLU across vision and language tasks, and BERT and GPT adopted it. Later models such as LLaMA moved to SwiGLU, a gated unit built on Swish.

Related explainers

References

  1. Hendrycks & Gimpel (2016), "Gaussian Error Linear Units (GELUs)"
  2. Ramachandran, Zoph & Le (2017), "Searching for Activation Functions"
  3. Clevert, Unterthiner & Hochreiter (2015), "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)"
  4. He et al. (2015), "Delving Deep into Rectifiers"
  5. Goodfellow, Bengio & Courville (2016), "Deep Learning", section 6.3: Hidden Units

Last updated