How Activation Functions Work
Why Not Just Linear?
Without activation, layers collapse into one
Two separate linear layers: Layer 1 and Layer 2. Press Compose layers to see what happens when we pass Layer 1's output into Layer 2.
Step 1 of 6: Why Not Just Linear?
Without activation, layers collapse into one
Two separate linear layers: Layer 1 and Layer 2. Press Compose layers to see what happens when we pass Layer 1's output into Layer 2.
Step 2 of 6: The Sigmoid
Squashing everything between 0 and 1
The sigmoid squashes any input to (0, 1). But in the saturated regions (far left/right), the gradient nearly vanishes — making learning painfully slow.
Step 3 of 6: ReLU — Keep It Simple
Zero if negative, pass if positive
ReLU is beautifully simple: pass positive values through, zero out negatives. The gradient is always 0 or 1 — no vanishing! But neurons with only negative inputs become permanently dead.
Step 4 of 6: Tanh — Centered Sigmoid
Output centered around zero for faster learning
Tanh is a zero-centered version of sigmoid, ranging from −1 to +1. This means its outputs are roughly balanced, which helps gradients flow in both directions and often makes training faster.
Step 5 of 6: The Activation Zoo
Leaky ReLU, ELU, Swish, and friends
Beyond ReLU, researchers have designed many variants. Leaky ReLU fixes dead neurons, Swish and GELU are smooth and used in modern transformers. Each trades off simplicity, speed, and gradient flow.
Step 6 of 6: Activations in Action
Watch non-linearity create decision boundaries
Without activations, the network can only draw a straight line as the decision boundary. It misclassifies 33 points (marked ✕). Toggle the activation on to see how a curved boundary separates the clusters much better.