ENID
Lensa ML

How Logs & Exponentials Work

Step 1 of 6: Exponential Growth

The function that grows as fast as itself

x value1.00
xeˣ(1.0, 2.72)
x
1.0
exp(x)
2.72
Slope
2.72

For positive x, growth speeds up. The slope (yellow dashed) still equals the value, so the larger exp(x) gets, the faster it grows.

Drag the x value slider from −3 to 3 and compare the exp(x) and SLOPE cards. They always match, because the derivative of eˣ is eˣ. That property defines e: among all bases b, only e gives a curve whose slope at x = 0 is exactly 1. Other bases differ by a constant, since bˣ = e^(x·ln b). At x = 3 the value is about 20.1, and at x = 10 it would be about 22,000, which is why exponentials overflow quickly in floating point.

Step 2 of 6: The Natural Logarithm

The inverse of exp — turning multiplication into addition

x value2.72
y = xeˣln(x)
x
2.72
ln(x)
1.001
exp(ln(x))
2.72

ln(2.7) = 1.00, and ln grows slowly. The two curves are mirror images across y = x because ln is the inverse of exp.

Move the x value slider between 0.1 and 10 and check that exp(ln(x)) returns x. The log is only defined for positive inputs, and it heads to minus infinity as x approaches 0. Logs in other bases are rescaled natural logs: log₂(x) = ln(x)/ln 2. Information theory uses base 2 and measures in bits; most ML code uses base e and measures in nats. One nat is about 1.44 bits.

Step 3 of 6: Multiplication → Addition

Why logarithms turn products into sums

a3.00
b4.00
Linear scale3.0a4.0b12.0a × bLog scale01.10log(a)1.39log(b)2.48log(a)+log(b)
a × b
12.0
log(a)
1.099
log(b)
1.386
log(a)+log(b)
2.485

log(3.0) + log(4.0) = 2.48 = log(12.0). Logarithms turn multiplication into addition, and sums are numerically stable and easy to differentiate.

Set a and b with the two sliders and compare log(a) + log(b) with the a × b card. The identity log(ab) = log(a) + log(b) is why slide rules worked and why log scales show multiplicative changes as equal steps. Two related rules follow: log(a/b) = log(a) − log(b) and log(aᵏ) = k·log(a). In ML these let a product of per-example probabilities become a sum of per-example losses, so a mini-batch gradient is an unbiased estimate of the full gradient.

Step 4 of 6: Logs in Cross-Entropy

Why wrong confident predictions are punished severely

Predicted probability p0.70
p−log(p)0.250.50.751danger zone−log(0.70) = 0.36
p
0.70
−log(p)
0.357
Penalty
LOW

At p = 0.70, the loss is 0.36. The model is uncertain, and the penalty pushes it toward the right answer.

Drag the Predicted probability p toward 0.01 and watch −log(p) rise. At p = 0.9 the loss is 0.105, at p = 0.5 it is 0.693, and at p = 0.01 it is 4.61. The penalty has no upper limit, so a single confident mistake can outweigh many correct answers. For a model that outputs probabilities through softmax, the gradient of this loss with respect to each logit is simply p minus the target, which makes training stable.

Step 5 of 6: Exponentials in Softmax

From raw logits to probabilities that sum to 1

Logit A2.00
Logit B1.00
Logit C-0.50
Temperature T1.00
Raw logitsA2.0B1.0C-0.5↓ softmax(z / T) ↓ProbabilitiesA69.0%B25.4%C5.7%
Temp
1.0
Max prob
69.0%
Entropy (bits)
1.106

Near the standard T = 1, a gap of d between two logits becomes a probability ratio of exp(d/T). Normalizing makes the probabilities sum to 1.

Adjust the three logits and the Temperature T. With the starting logits 2, 1, and −0.5 at T = 1, the probabilities are about 69%, 25%, and 6%. Adding the same constant to every logit changes nothing, since it cancels in the ratio. Libraries use that fact for numerical safety: they subtract the largest logit before exponentiating, so no exp call overflows. The same trick, called log-sum-exp, computes log(Σ exp(zᵢ)) without overflow.

Step 6 of 6: Why We Take Logs

Log-likelihood keeps numbers stable as data grows

Observations N8
Parameter p (heads probability)0.60
Data (coin flips)HTHHHHHHLog-likelihoods (each observation)Sum = log-likelihood-4.492
Product (likelihood)
0.60H × … × 0.60H × 0.40T × … × 0.40T
= 0.607 × 0.401 = 0.011197
Sum (log-likelihood)
log(0.60) + … + log(0.60) + log(0.40) + … + log(0.40)
= 7×(-0.511) + 1×(-0.916) = -4.492
Likelihood
0.0112
Log-likelihood
-4.492
MLE heads/N
0.88

With 8 observations, the likelihood is already 0.011197 , while the log-likelihood -4.49 is an ordinary number.

Raise Observations N and move the heads probability p to find where the log-likelihood peaks. The coin flips stay fixed while p moves, because p belongs to the model, not the data. The peak sits at p = heads/N, the maximum likelihood estimate, and the log does not move it because log is strictly increasing. At the default N = 8 the data has 7 heads, so the peak is near p = 0.88. In 64-bit floating point the smallest positive value is about 5 × 10⁻³²⁴. At p = 0.5, the likelihood of 1,075 coin flips is 2⁻¹⁰⁷⁵, which is smaller still and rounds to zero.

What are logarithms and exponentials in machine learning?

The exponential function exp(x) = eˣ raises Euler's number e ≈ 2.718 to the power x, and the natural logarithm ln(x) is its inverse: ln(eˣ) = x. Exponentials turn sums into products and grow faster than any polynomial. Logarithms turn products into sums and compress huge ranges of numbers into small ones.

Both appear throughout machine learning. Softmax and sigmoid use exponentials to turn raw scores into probabilities. Cross-entropy loss, log-likelihood, KL divergence, and perplexity all use logarithms. Log scales are also standard for learning-rate searches, and models often predict log-variance or log-odds because those quantities can be any number, positive or negative.

Key ideas

  • exp(x) is its own derivative, and it is always positive.
  • ln(x) is the inverse of exp(x) and is defined only for x > 0.
  • Logarithms turn products into sums: log(ab) = log(a) + log(b).
  • Cross-entropy loss is −log of the probability assigned to the correct answer.
  • Softmax exponentiates logits and normalizes them; temperature controls how sharp the result is.
  • Working in log space keeps likelihoods of large datasets from underflowing to zero.

Common questions

Why is e the base used in machine learning?

Because the derivative of eˣ is eˣ and the derivative of ln(x) is 1/x, with no extra constants. Any other base adds a factor of ln(b) to every gradient. The choice does not change which parameters are optimal, only the units of the loss.

What does log loss mean?

Log loss is another name for binary cross-entropy: −[y·log(p) + (1 − y)·log(1 − p)], averaged over examples. A model that always predicts 0.5 scores ln 2 ≈ 0.693, which is a useful baseline. Lower is better, and a perfect model scores 0.

Why do models predict log-variance instead of variance?

Variance must be positive, but a network's output can be any real number. Predicting s = log σ² and using σ² = exp(s) guarantees a positive variance without a constraint. Variational autoencoders and many uncertainty models use this parameterization.

What is the relationship between sigmoid and exp?

The sigmoid is σ(x) = 1/(1 + e⁻ˣ), and it equals a two-class softmax with logits x and 0. Its inverse is the logit, ln(p/(1 − p)), the log-odds of p. Logistic regression models these log-odds as a linear function of the features.

Related explainers

References

  1. Goodfellow, Bengio & Courville, "Deep Learning", ch. 3 Probability and Information Theory
  2. Deisenroth, Faisal & Ong, "Mathematics for Machine Learning"
  3. Zhang et al., "Dive into Deep Learning", section 4.1 Softmax Regression

Last updated