ENID
Lensa ML

Model Evaluation & Metrics

Step 1 of 6: Beyond Accuracy

Why accuracy isn't enough

Predicted 1Predicted 0Actual 1Actual 0TP50FN0FP50TN0
Class balance (% positive)50% / 50%
Accuracy
50%
Strategy
Always "1"
Useful?
No!

Near balance, the "always predict majority" trick scores only as high as the majority's share. As imbalance grows, accuracy becomes a misleading metric. Try dragging toward 90%.

Drag the class balance slider from 50% toward 5% or 95% and watch the accuracy card climb while the model stays useless. The baseline here predicts one class for every input, so its accuracy always equals the majority's share of the data. Compare any model against that baseline before trusting its accuracy. If 1 in 1,000 transactions is fraud, a model that flags nothing scores 99.9%. Balanced accuracy, the mean of the recall on each class, gives the same do-nothing model 50% at every level of imbalance.

Step 2 of 6: The Confusion Matrix

True/false × positive/negative

Predicted +Predicted −Actual +Actual −TP38FN12FP15TN35
Decision thresholdθ = 0.50
TP
38
FP
15
TN
35
FN
12

TP (correct positive) and TN (correct negative) are good. FP (false alarm) and FN (missed detection) are errors. Middle threshold: FP and FN errors are both moderate.

Drag the decision threshold between 0.05 and 0.95 and watch counts move between cells. The 100 examples are 50 positives and 50 negatives, so each row always sums to 50. Lowering the threshold can only move positives from FN to TP and negatives from TN to FP. Which error costs more depends on the task: a missed tumor (FN) is worse than an extra biopsy (FP), while a spam filter that hides a real email (FP) is worse than one spam message getting through.

Step 3 of 6: Precision & Recall

The tradeoff

RecallPrecision0.250.250.50.50.750.75
Threshold0.50
Precision
0.717
Recall
0.760
Threshold
0.50

Precision = "Of those I predicted positive, how many were correct?" (38/53). Recall = "Of all actual positives, how many did I find?" (38/50). Moving the threshold raises one and lowers the other.

Slide the threshold and follow the yellow dot along the precision-recall curve. At θ = 0.50 this model has precision 0.717 (38 of 53 flagged) and recall 0.760 (38 of 50 found). Recall ignores negatives entirely, but precision depends on class balance: if positives were ten times rarer, the same threshold would flag far more negatives per positive and precision would drop. That is why precision-recall curves are preferred over ROC curves when positives are rare. Average precision, the area under this curve, is the basis of the mAP score used in object detection.

Step 4 of 6: F1 Score

The harmonic mean

ThresholdScore000.50.511F1 peakF1PrecisionRecall
Threshold0.50
F1 score
0.738
Precision
0.717
Recall
0.760

The F1 score is the harmonic mean of precision and recall: 2PR/(P+R) = 0.738. The peak F1 = 0.794 is at θ ≈ 0.35. Try moving the threshold there.

Move the threshold toward the F1 peak marker. For these 100 scores F1 peaks at about 0.794 at θ = 0.35, where recall is 1.0 and precision is 0.658. The harmonic mean punishes a gap between the two: precision 1.0 with recall 0.1 gives F1 = 0.18, while the plain average would report 0.55. F1 also treats a false positive and a false negative as equally bad. When they are not, the Fβ score weights recall β times as heavily as precision, so F2 is common in screening, where misses are expensive.

Step 5 of 6: ROC Curve

True positive rate vs false positive rate

FPRTPR0.250.250.50.50.750.75randomAUC = 0.874
Threshold (sweeps the curve)0.50
TPR
0.760
FPR
0.300
AUC
0.874

The ROC curve plots TPR vs FPR at every threshold. The AUC (area under curve) = 0.874 summarizes all thresholds in one number. AUC = 1.0 is perfect, 0.5 is random. This model performs well above random.

Sweep the threshold and watch the dot travel the ROC curve, from the top right (flag everything) to the bottom left (flag nothing). AUC has a direct reading: it is the probability that a randomly chosen positive gets a higher score than a randomly chosen negative. That makes it independent of any single threshold and of class balance. The second property can mislead when positives are rare, because a small false positive rate can still mean thousands of false alarms. ROC analysis began with radar signal detection in the Second World War.

Step 6 of 6: Regression Metrics

MSE, MAE, R²

Model complexity (polynomial degree)degree = 1
MSE
1.24
MAE
0.87
R²
0.019

MSE penalizes large errors heavily. MAE is less sensitive to outliers. R² = 0.019 is the fraction of variance explained. A straight line underfits this curved data. Try a higher degree.

Drag the polynomial degree from 1 to 5. MSE squares each residual, so one error of 4 counts as much as sixteen errors of 1. MAE counts them as 4 and 16. R² = 1 − SS_res / SS_tot compares the fit with always predicting the mean, so R² = 0 means no better than that constant. Here the straight line scores only 0.019 on this hump, the quadratic 0.870, and degree 5 reaches 0.952 by chasing noise. A least-squares fit with an intercept never scores below 0 on its own training data, but R² can go negative on new data. All three readouts use the same 10 points the curve was fit to, so none of them measures performance on new data.

What are model evaluation metrics?

Model evaluation metrics are numbers that score how well a model's predictions match the true answers on data it was not trained on. For classification, most of them come from the confusion matrix: accuracy, precision, recall, F1, and the ROC curve with its area (AUC). For regression, the usual ones are mean squared error (MSE), mean absolute error (MAE), and R².

The metric decides which model gets shipped. Fraud detection, medical screening, and spam filtering have rare positives and unequal error costs, so accuracy alone hides the failures that matter. Ranking and recommendation care about ordering, which AUC measures. Forecasting teams choose between MSE and MAE depending on whether a few large misses are worse than many small ones. The same metrics also set decision thresholds, compare candidate models, and trigger early stopping during training.

Key ideas

  • Accuracy is misleading on imbalanced data; always compare against the majority-class baseline.
  • The confusion matrix (TP, FP, TN, FN) is the source of every classification metric.
  • Precision asks how many flagged cases are real; recall asks how many real cases were flagged.
  • Moving the decision threshold trades precision against recall, and F1 summarizes the balance.
  • ROC AUC is the probability that a random positive outranks a random negative.
  • MSE punishes large errors more than MAE, and R² compares the model against predicting the mean.

Short takes on this topic

Common questions

When should I use PR AUC instead of ROC AUC?

Use the precision-recall curve when positives are rare and false alarms are what you care about, as in fraud or rare-disease detection. ROC AUC can stay high in those settings because the large pool of negatives keeps the false positive rate small. A random classifier has ROC AUC 0.5, but its PR AUC equals the positive rate, so PR AUC must be read against that baseline.

What is the difference between macro and micro averaging?

For multi-class problems, macro averaging computes a metric such as F1 for each class and then takes the plain mean, so rare classes count as much as common ones. Micro averaging pools all TP, FP, and FN counts first, so frequent classes dominate. In single-label multi-class classification, micro-averaged F1 equals accuracy.

What is a good AUC score?

There is no universal cutoff, because it depends on how hard the task is and what the alternatives score. 0.5 is chance and 1.0 is perfect ranking. Compare AUC against a simple baseline and against existing models on the same test set rather than against a fixed number.

Should I report RMSE or MSE?

RMSE is the square root of MSE, so it is in the same units as the target, which makes it easier to read. The two rank models identically. MSE is more common as a training loss because its gradient is simpler.

Related explainers

References

  1. Fawcett (2006), "An introduction to ROC analysis", Pattern Recognition Letters
  2. Davis & Goadrich (2006), "The Relationship Between Precision-Recall and ROC Curves", ICML
  3. Saito & Rehmsmeier (2015), "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets", PLoS ONE
  4. James, Witten, Hastie & Tibshirani, "An Introduction to Statistical Learning"

Last updated