Part 2
Classic ML evaluation is the foundation everything else builds on. This part covers the standard metrics for classification, regression, and ranking/retrieval tasks, plus the analysis practices — error analysis, slice evaluation, and robustness testing — that turn raw numbers into insight.
For binary classification, each example is either positive or negative, and each prediction is right or wrong. That gives four outcomes: true positive (TP), true negative (TN), false positive (FP), and false negative (FN).
Accuracy is the fraction of predictions that were correct:
accuracy = (TP + TN) / (TP + TN + FP + FN)It is the easiest metric to understand, but it hides how the errors are distributed. If 95% of your data is negative, a model that always predicts "negative" scores 95% accuracy while learning nothing.
Precision answers "of everything the model called positive, how much was actually positive?"
precision = TP / (TP + FP)Recall answers "of all the real positives, how many did the model find?"
recall = TP / (TP + FN)They trade off against each other. Raising the decision threshold usually increases precision but lowers recall, and vice versa. Which one matters depends on the cost of each error type. For spam filters, false positives (deleting good mail) are worse than false negatives, so precision matters more. For cancer screening, false negatives are far more dangerous, so recall dominates.
F1 is the harmonic mean of precision and recall, giving a single number when you care about both:
F1 = 2 * (precision * recall) / (precision + recall)Because it is a harmonic mean, F1 is pulled down hard when either metric is low — a model at precision 0.9 and recall 0.1 gets F1 ≈ 0.18, not 0.5.
ROC-AUC measures how well the model ranks examples across every decision threshold. It plots the true positive rate against the false positive rate; the area under that curve is the probability that a random positive is ranked above a random negative. A coin flip scores 0.5; a perfect ranker scores 1.0.
Precision-recall curves focus on the positive class. The area under the PR curve is much more informative than ROC-AUC when positives are rare, because precision reflects how often a positive prediction is actually right. When class balance is extreme, prefer PR-AUC.
Regression models predict continuous numbers. Let the true value be y and the prediction be y_hat; the error is y - y_hat.
Mean Absolute Error is the average absolute distance between predictions and truth. It is expressed in the same units as the target, so it is easy to explain — "on average we are off by 3 units."
MAE = mean(|y - y_hat|)Root Mean Squared Error squares the errors before averaging, which penalizes large mistakes far more heavily than small ones.
RMSE = sqrt(mean((y - y_hat)^2))RMSE is always ≥ MAE; the gap grows as large errors appear. If RMSE is much larger than MAE, a few outliers are dominating your loss — decide whether that is the behavior you want.
R² measures how much of the variance in the target the model explains, relative to simply predicting the mean.
R² = 1 - SS_residual / SS_totalAn R² of 1 is a perfect fit, 0 means the model is no better than predicting the mean, and negative values mean it is worse. R² is convenient but can be inflated by the range of the data; it does not tell you whether the model is accurate in absolute terms.
For search and recommendation systems, what matters is where the good items appear in an ordered list of K results.
| Metric | What it measures | When to use it |
|---|---|---|
| Recall@K | Fraction of relevant items found inside the top K | How complete the shortlist is |
| Precision@K | Fraction of the top K that are relevant | How clean the shortlist is |
| MRR | Reciprocal rank of the first relevant item | When the user only needs one good hit (Q&A) |
| MAP | Average precision across queries, averaged again over queries | Overall ranking quality across the whole set |
| NDCG | Ranked usefulness with discounting for lower ranks | When relevance is graded, not binary |
Recall@K checks whether the important items made it into the top K: divide the number of relevant items in the top K by the total number of relevant items. Precision@K divides the relevant items in the top K by K. A high-precision shortlist with low recall means you found only a few of the good items; high recall with low precision means you returned lots of junk to be thorough.
Mean Reciprocal Rank looks only at the first correct answer. If the first correct result is at position 2, the reciprocal rank is 1/2. It is ideal for "single-answer" tasks where the user is satisfied by the first relevant hit.
Mean Average Precision computes average precision for each query (precision at every relevant position, averaged), then averages across all queries. It rewards systems that put relevant items both early and densely.
Normalized Discounted Cumulative Gain supports graded relevance. Relevant results get less credit the further down the list they sit (the discount), and the total is normalized against the ideal ordering so different queries are comparable. NDCG is the default choice for search evaluation with relevance grades.
ranx library. Write a small helper script to compute the full metric family on a held-out query set, and log the output with each experiment.Metrics tell you how much you are failing; error analysis tells you why. Sample misclassified examples, group them by pattern (e.g., "all failures involve negation," "all failures are very short texts"), and rank the groups by how many errors they explain. Fix the biggest groups first. Doing this before reaching for a bigger model often saves days of work.
Overall accuracy hides uneven performance. Split the data into slices — by language, geography, device, text length, customer segment — and compute metrics per slice. A model can average 95% accuracy while failing badly for one demographic. If a slice underperforms, that is usually a data problem in that slice, not a general modeling problem.
Real-world input is messy: typos, slang, rephrasing, adversarial formatting. Robustness testing feeds perturbed versions of the same inputs — misspelled words, shuffled tokens, synonyms, added noise — and checks that predictions stay stable. If accuracy collapses under mild perturbation, the model is memorizing surface patterns instead of understanding.