ENID
Lensa ML

How Loss Functions Work

Step 1 of 6: Prediction vs Reality

Networks guess — loss measures how wrong they are

Predicted0.60Actual0.85error: 0.25
Adjust the predictionpredicted = 0.60
Error
0.250
Loss (squared)
0.0625

A loss function measures how far the network's prediction is from the true value. Here the true value is 0.85. Watch the error and its square shrink as the prediction moves toward it.

Drag the prediction slider from 0 to 1. At the starting prediction of 0.6, the error is 0.25 and the squared loss is 0.0625. Terminology varies, but the loss usually refers to one example and the cost to the average over a dataset or batch; the optimizer minimizes the cost. The squared loss reaches zero only when the prediction matches 0.85 exactly. Its slope is also zero there, so gradient-based training slows down smoothly as the prediction closes in.

Step 2 of 6: Mean Squared Error

Squaring mistakes so big errors hurt more

xy0.10.40.40.51.00.41.3y = 1.30x + 0.80
MSE = (1/n) Σ (yᵢ − ŷᵢ)²
= (0.1² + 0.4² + 0.4² + 0.5² + 1.0² + 0.4² + 1.3²) / 7
= (0.0 + 0.2 + 0.2 + 0.3 + 1.0 + 0.2 + 1.7) / 7
= 3.43 / 7 = 0.490
Slope1.30
Intercept0.80

MSE squares each residual so that large errors are penalized much more than small ones. The pink squares draw each squared error as an area. Adjust the slope and intercept to shrink the total area.

Drag Slope (0.30 to 2.20) and Intercept (−3 to 5) to shrink the squares. The starting line, 1.3x + 0.8, has an MSE of 0.490. The best possible line for these seven points is about 1.45x + 0.79, with an MSE of 0.057. Because errors are squared, one residual of 10 adds as much as one hundred residuals of 1, so a single outlier can drag the whole line. MSE is the natural loss when noise is Gaussian: minimizing it is the same as maximum likelihood under that assumption.

Step 3 of 6: Binary Cross-Entropy

The loss function for yes-or-no decisions

predicted probability (p)loss— -log(p)— -log(1-p)loss = 0.357
Predicted probabilityp = 0.70

Binary cross-entropy uses -log(p) when the label is 1 and -log(1-p) when it's 0. A confident wrong prediction produces a huge loss that pushes the network to correct itself.

Switch the true label between 1 and 0 and drag the predicted probability. The full formula is L = −[y log p + (1 − y) log(1 − p)], where only one term is active for each label. At p = 0.7, the loss is 0.357 when the label is 1 and 1.204 when it is 0. At p = 0.01 with label 1 it reaches 4.61. Frameworks usually compute it from the raw logit, as in PyTorch's BCEWithLogitsLoss, which avoids taking log(0) when a sigmoid saturates.

Step 4 of 6: Categorical Cross-Entropy

Extending to multiple classes

LogitsSoftmaxProbabilitiesCat2.51.0 / 1.40.73673.6%← trueDog1.00.2 / 1.40.16416.4%Bird0.50.1 / 1.40.10010.0%Loss = −log(p_true)= −log(0.7361)= 0.3064
Cat logit2.5
Dog logit1.0
Bird logit0.5

Categorical cross-entropy only cares about the probability assigned to the true class. The loss is -log(p_true), so how the rest of the probability splits among the wrong classes doesn't matter.

Pick the true class with the True buttons and move the three logit sliders. The starting logits 2.5, 1.0, and 0.5 give Cat 73.6%, Dog 16.4%, and Bird 10.0%. The loss is 0.306 if Cat is correct, 1.806 for Dog, and 2.306 for Bird. A model that guesses uniformly over K classes has a loss of ln K, about 1.10 here and 6.91 for ImageNet's 1,000 classes. That makes a quick sanity check for an untrained model. Label smoothing replaces the one-hot target with, say, 0.9 for the true class.

Step 5 of 6: The Loss Landscape

Loss as a surface over the weights

w1w2min
Loss:
<0.5
0.5–1.5
1.5–3
3–5
5–8
8–12
12–18
≥18
w13.00
w22.50
Loss
13.13
Gradient
6.67

The loss landscape is a terrain where every point represents a set of weights, and the height is the loss. The yellow arrow shows the direction of steepest descent, where gradient descent moves next.

Drag w1 and w2 between −3.5 and 3.5. This surface is L = (w1 − 1)² + (w2 + 0.5)² + 0.5 sin(2w1) sin(2w2): a bowl centered at (1, −0.5) with ripples on top. The starting point (3, 2.5) has a loss of 13.13. The lowest point is near (0.9, −0.64), where the loss is about −0.44. The sine term lets it dip below zero, which MSE and cross-entropy never do. A real network's loss surface has millions of dimensions and can only be plotted along one or two chosen directions.

Step 6 of 6: Loss Guides Learning

Gradients flow from loss back through the network

1. Forward Pass
2. Compute Loss
3. Backward Pass
LInputHiddenOutputLossdata flows forward →

Forward pass: data flows through the network to produce a prediction. The loss measures the error, then gradients flow backward, telling each weight how to adjust. This cycle repeats thousands of times during training.

Watch the animation cycle through Forward Pass, Compute Loss, and Backward Pass. The loss being minimized often has more than one part. Weight decay, for example, adds λ‖w‖² to the data loss, so the model pays for large weights as well as wrong answers. The loss used for training also differs from the metric people report: a classifier minimizes cross-entropy but is judged on accuracy or F1. Tracking the loss on held-out validation data shows when the model starts to overfit.

What is a loss function in machine learning?

A loss function measures how wrong a model's prediction is, as a single number. It compares the prediction with the correct value and returns 0 for a perfect answer and larger values for worse ones. Training a model means adjusting its parameters to make the average loss over the training data as small as possible.

The loss is the only signal the optimizer sees, so choosing it decides what the model learns to care about. Regression models typically use mean squared error or mean absolute error. Classifiers use binary or categorical cross-entropy, and language models train on cross-entropy over the next token. Specialized tasks add their own losses, such as focal loss for object detection or contrastive losses for embeddings.

Key ideas

  • A loss function turns the gap between prediction and truth into one number.
  • Training adjusts the parameters to minimize the average loss over the data.
  • MSE squares errors, so large errors and outliers dominate it.
  • Cross-entropy is −log of the probability given to the correct class.
  • Confident wrong predictions get very large cross-entropy losses.
  • The gradient of the loss tells each weight which way to move.

Common questions

Why not train directly on accuracy?

Accuracy changes in jumps: a small change to a weight usually flips no predictions, so its gradient is zero almost everywhere. Gradient descent gets no direction from it. Cross-entropy changes smoothly with every probability, and lowering it tends to raise accuracy.

When is MAE a better choice than MSE?

Use MAE when the data has outliers that should not dominate training. Minimizing MSE predicts the mean of the target, while minimizing MAE predicts the median, which outliers barely move. Huber loss behaves like MSE for small errors and like MAE for large ones.

Which loss function works for imbalanced classes?

Weighted cross-entropy multiplies the loss of each class by a weight, often inversely proportional to how common the class is. Focal loss scales down the loss on examples the model already gets right, so training concentrates on the rare, hard ones. It was introduced for object detection, where background boxes far outnumber objects.

Why does training loss go down while validation loss goes up?

The model is overfitting: it is memorizing details of the training examples that do not carry over to new data. Common fixes are stopping training at the lowest validation loss, adding regularization such as weight decay or dropout, and collecting more data or using data augmentation.

Related explainers

References

  1. Goodfellow, Bengio & Courville (2016), "Deep Learning", section 6.2.1: Cost Functions
  2. Li et al. (2018), "Visualizing the Loss Landscape of Neural Nets"
  3. Szegedy et al. (2016), "Rethinking the Inception Architecture for Computer Vision" (label smoothing)
  4. Lin et al. (2017), "Focal Loss for Dense Object Detection"

Last updated