Why 99% accuracy can mean a useless model
Accuracy is the share of a model's predictions that are correct. When one class is rare, a model can score 99% and still do nothing useful. Fraud detection is the classic case. Take 10,000 card transactions, of which 100 are fraud, a 1% rate.
2 min read · Lensa ML
- A confusion matrix counts a model's calls by the actual class. Rows are the truth, normal or fraud, and columns are the call, passed or flagged. A true negative (TN) is a normal transaction passed, a false positive (FP) is a normal one flagged, a false negative (FN) is a fraud passed, and a true positive (TP) is a fraud flagged. The top model always says normal, so it passes all 10,000: 9,900 TN and 100 FN.
- Its accuracy is (TP + TN) / 10,000 = 9,900 / 10,000 = 99%. Recall, the share of fraud that gets flagged, is TP / (TP + FN) = 0 / 100 = 0%. Precision, the share of flagged transactions that are fraud, is TP / (TP + FP) = 0 / 0. The model flags nothing, so precision is undefined. F1, a single score that combines precision and recall, is then reported as N/A or as 0, depending on the convention. The explainer shows both as N/A.
- The bottom model catches 80% of fraud and wrongly flags 4% of normal transactions: 80 TP, 20 FN, 396 FP and 9,504 TN. Its accuracy is lower, 95.8%, because it makes 416 mistakes, 396 false alarms and 20 misses, against the first model's 100. Yet it finds 80 of the 100 frauds. Recall is 80%, precision 16.8% and F1 27.8%. Accuracy ranks it below the model that catches nothing.
Takeaway
On imbalanced data, compare every model with the always-majority baseline, whose accuracy equals the majority share, here 99%. Report recall, precision and F1 on the rare class instead of accuracy alone. Then pick the trade-off from what each mistake costs: a missed fraud usually costs far more than a false alarm sent for review.
Go deeper · interactive
Class Imbalance & Resampling
On step 2, drag the Minority class % slider and watch accuracy fall while recall stays at 0%.
Open the explainer