Deep Learning

A classifier can be right most of the time and still be wrong in the way that matters. Evaluation is the practice of finding out which kind of wrong you have, and whether the cost of that error is acceptable. These notes focus on binary classification, where every prediction is positive or negative.

The confusion matrix

Four counts sit underneath most classification metrics:

Every metric below is a different way of combining those four numbers. The choice of metric is a statement about which mistakes you care about.

Accuracy and when it misleads

Accuracy is the share of all predictions that are correct: true positives plus true negatives, divided by the total number of examples. It is easy to explain and easy to compare across models, which is why it is often the first number reported.

The problem appears when the classes are unbalanced. If positive cases are rare, a model that always predicts negative can look accurate while missing every case you care about. A single number cannot tell you whether the model has learned anything useful.

Precision and recall

Precision answers a narrow question: of the examples the model called positive, how many were actually positive? Recall asks the mirror question: of the examples that were actually positive, how many did the model find?

The two pull against each other. Lowering the threshold for a positive prediction usually finds more true positives, but it can also add false positives. Raising the threshold does the opposite. Which direction is better depends on the cost of a false alarm versus the cost of a miss.

The F1 score

F1 combines precision and recall into one number using the harmonic mean. It is useful when you want a single summary that balances the two, and when both kinds of error matter. It is less useful when one kind of error is clearly more expensive, because the balance it assumes may not match the problem.

F1 can also be computed per class. If one class is the reason the model exists, report the score for that class rather than hiding it inside a global average.

ROC and AUC

A receiver operating characteristic curve plots the true positive rate against the false positive rate as the decision threshold moves. Each point on the curve is a different trade between catching positives and raising false alarms. The area under the curve, usually called AUC, summarises the curve into one number.

AUC is useful for comparing ranking behaviour across thresholds, but it can be optimistic when the positive class is very rare. For imbalanced problems, a precision recall curve often tells a more honest story because it focuses on the positive class.

A short checklist

• Decide which error is more expensive before choosing a metric.
• Report more than one number, especially when the classes are unbalanced.
• Look at the confusion matrix, not only the summary.
• Check performance per class and per important subgroup.
• Write down the threshold you chose and why.