Lensa ML
Lensa ML

How Softmax Works

Step 1 of 6: Raw Logits

Before softmax

z₁2.0
z₂1.0
z₃-1.0
02.0z₁1.0z₂-1.0z₃5−5logit value

z₁

2.0

z₂

1.0

z₃

-1.0

z₁ is the largest logit, so the model favors class 1. Raw logits are not probabilities yet.

Drag z₁, z₂, and z₃ anywhere from −5 to 5. Logits are the raw outputs of a network's last layer, one per class, and nothing constrains their sign or scale. The word comes from statistics, where the logit of a probability p is the log-odds ln(p / (1 − p)). Softmax never changes the ranking, so the class with the largest logit always gets the largest probability. A language model such as GPT-2 produces 50,257 logits at every position, one per token in its vocabulary.

Step 2 of 6: The Softmax Function

eᶻ / Σeᶻ

z₁2.0
z₂1.0
z₃-1.0
Raw Logits02.0z₁1.0z₂-1.0z₃→eᶻ / ΣeᶻProbabilities70.5%P₁25.9%P₂3.5%P₃

exp(z₁)

7.389

+

exp(z₂)

2.718

+

exp(z₃)

0.368

=

Σeᶻ

10.475

Probabilities are spread across the classes. They still sum to 1.0, as softmax outputs always do.

Move the three logits and watch the bars on the right. The formula is P(i) = e^(zᵢ) / Σⱼ e^(zⱼ). For the starting logits 2, 1, and −1, it gives 0.705, 0.259, and 0.035. Only differences between logits matter: adding the same constant to all three leaves the output unchanged, and a gap of 1 makes one probability e ≈ 2.72 times another. The exp readouts start at 7.389, 2.718, and 0.368, which sum to 10.475. Dividing each by that sum gives the probabilities.

Step 3 of 6: Temperature

Sharpening & smoothing

z₁2.0
z₂1.0
z₃-0.5
Temperature (T)1.00
Raw Logits02.0z₁1.0z₂-0.5z₃→eᶻᐟᵀ / ΣeᶻᐟᵀProbabilities69.0%P₁25.4%P₂5.7%P₃T=1.00 — standard

Temperature

1.00

Max P

69.0%

Entropy

1.106

T = 1 is standard softmax, with no temperature scaling.

Drag Temperature (T) from 0.1 to 5. Temperature divides every logit before softmax: P(i) = e^(zᵢ/T) / Σⱼ e^(zⱼ/T). For these logits the top class gets 69.0% at T = 1, 87.6% at T = 0.5, and 41.2% at T = 5. The entropy readout is in bits; with three classes its maximum is log₂ 3 ≈ 1.585, reached when all classes are equal. Knowledge distillation uses a high T to expose a teacher model's soft preferences, and temperature scaling uses a fitted T to calibrate a classifier's confidence.

Step 4 of 6: Numerical Stability

The max trick

z₁100
The max trick— subtract max(z) before exp to prevent overflow
zᵢzᵢ − maxexp(·)P
z₁100→0.0→1.000→70.5%
z₂99→-1.0→0.368→25.9%
z₃97→-3.0→0.050→3.5%
Naive exp(zᵢ)70.5%P₁25.9%P₂3.5%P₃max trick→Stable exp(zᵢ−max(z))70.5%P₁25.9%P₂3.5%P₃

max(z)

100

Naive exp(z₁)

2.7e+43

Stable exp(z₁−max)

1.000

exp(100) = 2.7e+43, already huge. Shifting the logits so the largest is 0 keeps every exp value in a safe range, with the same result.

Slide z₁ from 0 to 1000; the other two logits stay 1 and 3 below it. In 64-bit floats, exp overflows once its input passes about 709.8, so the naive version breaks at 710. In 32-bit floats it breaks above about 88.7, and in 16-bit floats above about 11.1, which real logits easily reach. The trick is exact: e^(zᵢ − m) / Σⱼ e^(zⱼ − m) multiplies the top and bottom by the same e^(−m). Frameworks go one step further and fuse softmax with the log in cross-entropy, computing log-softmax via the log-sum-exp trick.

Step 5 of 6: Softmax vs Sigmoid

Multi-class vs binary

Sigmoid (binary)
σ(z) = 1 / (1 + e⁻ᶻ)
Softmax (multi-class)
P(i) = eᶻⁱ / Σeᶻʲ
z1.5
0.51.081.8%Yes18.2%NoSigmoid: P(Yes) vs P(No) = 1 − σ(z)σ(z)0.51.076.6%C₁17.1%C₂6.3%C₃Softmax: probability split across 3 classesP(C₁)P(C₂)P(C₃)

Sigmoid σ(z)

0.8176

Softmax P(C₁)

0.7662

Σ softmax

1.0000

Sigmoid gives a single P(Yes). Softmax splits probability across 3 classes, so P(C₁) is lower than σ(z).

Drag z from −5 to 5 and compare σ(z) with the softmax over the three logits [z, 0, −1]. With two classes, softmax reduces exactly to a sigmoid: softmax([z₁, z₂])₁ = σ(z₁ − z₂). So a binary classifier can use one sigmoid output or a two-way softmax, and both give the same model. At z = 1.5, the sigmoid gives 0.818 and the three-class softmax gives 0.766. Use independent sigmoids when several labels can be true at once, such as tags on a photo.

Step 6 of 6: Softmax in Practice

Classification pipeline

feat₁0.80
feat₂-0.30
feat₃0.50
f10.8f2-0.3f30.5z11.26z2-1.19z30.80softmaxeᶻ/Σeᶻ58.2%Cat5.0%Dog36.8%Bird★

Logit Cat

1.26

Logit Dog

-1.19

Logit Bird

0.80

Predicted

Cat

The model leans toward Cat but isn't fully confident. Move the feature sliders to see how logits and probabilities change.

Move feat₁, feat₂, and feat₃ between −2 and 2. Each logit is one row of the 3×3 weight matrix W dotted with the feature vector. The starting features [0.8, −0.3, 0.5] give logits 1.26 for Cat, −1.19 for Dog, and 0.80 for Bird, or 58%, 5%, and 37%. Training uses cross-entropy, −ln P(correct class); if Cat is correct, the loss is −ln 0.582 ≈ 0.54. Its gradient with respect to each logit is simply P(i) minus the target, 1 for the correct class and 0 for the rest.

What is the softmax function?

The softmax function turns a vector of raw scores, called logits, into a probability distribution. It exponentiates each score and divides by the sum of all the exponentials, so every output lies between 0 and 1 and the outputs add up to 1. Larger scores get larger probabilities, and the gap between scores sets how lopsided the result is.

Softmax is the standard output layer for multi-class classification, from digit recognition to ImageNet's 1,000 classes. Language models apply it over their whole vocabulary to pick the next token, and attention layers apply it to decide how much each token attends to the others. It is almost always paired with cross-entropy loss during training.

Key ideas

  • Softmax maps any vector of real numbers to probabilities that sum to 1.
  • It preserves the ranking of the logits, so the top logit is always the top class.
  • Only differences between logits matter; shifting them all by a constant changes nothing.
  • Temperature below 1 sharpens the distribution, and above 1 flattens it.
  • Subtracting the largest logit before exponentiating prevents overflow.
  • For two classes, softmax is the same as a sigmoid of the logit difference.

Short takes on this topic

Common questions

Why is it called softmax?

It is a smooth, differentiable version of the argmax function. Argmax puts all the weight on the largest score, while softmax gives most of the weight to it and some to the rest. As the temperature approaches 0, softmax approaches argmax.

Why does softmax use the exponential function?

The exponential is positive for every input, so negative logits still give valid probabilities. It also turns differences into ratios, which makes logits act as unnormalized log-probabilities. That pairs neatly with the logarithm in cross-entropy loss and gives the simple gradient P − y.

Are softmax outputs true probabilities?

They form a valid distribution, but they are not guaranteed to match how often the model is right. Large neural networks are often overconfident: a 95% softmax output may be correct much less than 95% of the time. Calibration methods such as temperature scaling correct this after training.

Where is softmax used besides the output layer?

Attention layers in transformers apply softmax to the scores between a query and every key, producing weights that sum to 1 over the sequence. Policy networks in reinforcement learning use it to turn action scores into a probability of taking each action.

Related explainers

References

  1. Goodfellow, Bengio & Courville (2016), "Deep Learning", section 6.2.2.3: Softmax Units for Multinoulli Output Distributions
  2. Zhang et al., "Dive into Deep Learning", Softmax Regression
  3. Hinton, Vinyals & Dean (2015), "Distilling the Knowledge in a Neural Network"
  4. Guo et al. (2017), "On Calibration of Modern Neural Networks"

Last updated