Model Evaluation Cheat Sheet
Confusion Matrix Β· Accuracy Β· Precision Β· Recall Β· F1 Β· AUC-ROC Β· RMSE Β· MAE Β· RΒ² Β· Learning Curves
Sheet 3 of 4
Machine Learning
Intermediate
Printable
Confusion Matrix
FoundationThe confusion matrix is the starting point for all classification metrics. It compares predicted labels vs actual labels across all classes.
Predicted: Positive
Predicted: Negative
Actual: Positive
TPTrue Positive
FNFalse Negative
Actual: Negative
FPFalse Positive
TNTrue Negative
| Term | Meaning | Also Called |
|---|---|---|
| TP | Predicted positive, actually positive | Hit |
| FP | Predicted positive, actually negative | Type I Error, False Alarm |
| FN | Predicted negative, actually positive | Type II Error, Miss |
| TN | Predicted negative, actually negative | Correct Rejection |
Memory trick: The first word (True/False) = was the prediction correct? The second word (Positive/Negative) = what did the model predict?
Classification Metrics
FormulasAccuracy
Accuracy = (TP + TN) / (TP + TN + FP + FN)
Overall correct predictions β misleading on imbalanced data
Precision
Precision = TP / (TP + FP)
Of all predicted positives, how many were actually positive? β minimises false alarms
Recall (Sensitivity)
Recall = TP / (TP + FN)
Of all actual positives, how many did we catch? β minimises misses
F1 Score
F1 = 2 Β· (Precision Β· Recall) / (Precision + Recall)
Harmonic mean β use when precision and recall both matter
Specificity (True Negative Rate)
Specificity = TN / (TN + FP)
Of all actual negatives, how many did we correctly identify?
PrecisionβRecall Tradeoff
Decision| Threshold β | Precision | Recall | Use When |
|---|---|---|---|
| High threshold | β Higher | β Lower | Cost of FP is high (spam filter, fraud flag) |
| Low threshold | β Lower | β Higher | Cost of FN is high (cancer screening, fault detection) |
| Use Case | Prioritise | Why |
|---|---|---|
| Cancer screening | Recall | Missing a case (FN) is far worse than a false alarm |
| Spam filter | Precision | Blocking legit email (FP) is worse than missing spam |
| Fraud detection | Recall | Missing fraud is costly; review team handles FPs |
| Search results | Precision | Users expect every result to be relevant |
| Balanced | F1 | Neither FP nor FN dominates |
FΞ² Score β Weighted F
# Ξ² > 1 β weight recall more (FN matters) # Ξ² < 1 β weight precision more (FP matters) FΞ² = (1+Ξ²Β²) Β· (PΒ·R) / (Ξ²Β²Β·P + R) # Common: F2 (recall 2Γ weight) sklearn: fbeta_score(y, Ε·, beta=2)
ROC Curve & AUC
Threshold-FreeROC Axes
X-axis: FPR = FP / (FP + TN)
Y-axis: TPR (Recall) = TP / (TP + FN)
| AUC Value | Interpretation |
|---|---|
| 1.0 | Perfect classifier |
| 0.9 β 1.0 | Excellent |
| 0.8 β 0.9 | Good |
| 0.7 β 0.8 | Fair |
| 0.6 β 0.7 | Poor |
| 0.5 | Random guess (no skill) |
| < 0.5 | Worse than random β flip predictions |
sklearn
from sklearn.metrics import ( roc_auc_score, roc_curve, average_precision_score) auc = roc_auc_score(y_test, y_prob) ap = average_precision_score(y_test, y_prob) fpr, tpr, thresholds = roc_curve(y_test, y_prob)
Imbalanced data: AUC-ROC can be misleading. Prefer PR-AUC (Average Precision) when the positive class is rare.
Regression Metrics
RegressionMAE β Mean Absolute Error
MAE = (1/n) Β· Ξ£ |yα΅’ β Ε·α΅’|
Same unit as target Β· robust to outliers Β· easy to interpret
MSE β Mean Squared Error
MSE = (1/n) Β· Ξ£ (yα΅’ β Ε·α΅’)Β²
Penalises large errors more heavily Β· differentiable β used in training
RMSE β Root Mean Squared Error
RMSE = βMSE
Same unit as target Β· most common reported metric
RΒ² β Coefficient of Determination
RΒ² = 1 β (SS_res / SS_tot)
1 = perfect Β· 0 = predicts mean only Β· can be negative (worse than mean)
| Metric | Outlier Sensitive? | Interpretable Unit? | Best For |
|---|---|---|---|
| MAE | β No | β Yes | Robust evaluation, reporting |
| RMSE | β Yes | β Yes | When large errors matter more |
| RΒ² | β Yes | β Unitless | Comparing models on same data |
| MAPE | β No | β % | Comparing across scales |
Multi-class Metrics
Multi-class| Averaging | How | Use When |
|---|---|---|
| Macro | Mean of per-class metrics (unweighted) | All classes equally important |
| Weighted | Mean weighted by class support (n) | Imbalanced classes β reflects real distribution |
| Micro | Aggregate TP/FP/FN across all classes | Every sample equally important |
sklearn β Classification Report
from sklearn.metrics import classification_report print(classification_report( y_test, y_pred, target_names=['cat','dog','bird'])) # Output per class: # precision recall f1-score support
Confusion Matrix β Multi-class
from sklearn.metrics import ConfusionMatrixDisplay import matplotlib.pyplot as plt ConfusionMatrixDisplay.from_estimator( model, X_test, y_test, cmap='Blues') plt.show()
Cohen's Kappa: Measures agreement correcting for chance. Kappa > 0.8 = strong Β· 0.6β0.8 = moderate Β· < 0.6 = poor.
Cross-Validation Strategies
Validation| Strategy | Best For | Cost |
|---|---|---|
| k-Fold (k=5,10) | General balanced datasets | Medium |
| Stratified k-Fold | Imbalanced classification | Medium |
| Leave-One-Out (LOO) | Very small datasets | Very High |
| Repeated k-Fold | Reducing variance of CV estimate | High |
| TimeSeriesSplit | Temporal / sequential data | Medium |
| Group k-Fold | Samples with group membership | Medium |
sklearn β Cross-Val Score
from sklearn.model_selection import ( cross_val_score, StratifiedKFold) cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scores = cross_val_score( model, X, y, cv=cv, scoring='f1_weighted') print(f"CV: {scores.mean():.3f} Β± {scores.std():.3f}")
Learning Curves
DiagnosisPlot train score vs val score as training set size increases β diagnoses bias vs variance.
| Pattern | Diagnosis | Fix |
|---|---|---|
| Train high, val low, large gap | Overfitting / High Variance | More data, regularise, reduce features |
| Both train & val low, small gap | Underfitting / High Bias | More complex model, more features |
| Both converge high | Good fit β | Deploy! |
| Val improves then plateaus | Need more data | Collect more training samples |
sklearn β Learning Curve
from sklearn.model_selection import learning_curve train_sz, train_sc, val_sc = learning_curve( model, X, y, train_sizes=np.linspace(.1, 1.0, 10), cv=5, scoring='f1') # Plot mean Β± std of train_sc and val_sc # vs train_sz to diagnose fit
Validation curve: Plot score vs a single hyperparameter value β shows optimal range before overfitting kicks in.
Which Metric to Use β Quick Reference
Decision Guide| Situation | Recommended Metric(s) | Why |
|---|---|---|
| Balanced binary classification | Accuracy, F1 | Classes roughly equal β accuracy not misleading |
| Imbalanced binary (rare positives) | PR-AUC, F1, Recall | Accuracy misleads β focus on positive class |
| Need probability calibration | Log Loss (Binary Cross-Entropy) | Measures quality of probability estimates directly |
| Ranking / threshold-free | AUC-ROC | Measures discriminatory power across all thresholds |
| Multi-class balanced | Macro F1, Accuracy | Treats all classes equally |
| Multi-class imbalanced | Weighted F1 | Accounts for class frequency |
| Regression β outliers matter | RMSE | Squares errors β penalises outliers more |
| Regression β outliers present | MAE, Huber | Robust β doesn't blow up on outliers |
| Regression β explained variance | RΒ² | Proportion of variance explained by model |
| Comparing across scales/datasets | MAPE, SMAPE | Percentage β scale-independent |
Never report only accuracy on imbalanced data. A model predicting the majority class always achieves high accuracy but has zero predictive value for the minority class.
sklearn Metrics Syntax
CodeClassification
from sklearn.metrics import ( accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, roc_auc_score) accuracy_score(y_test, y_pred) precision_score(y_test, y_pred) recall_score(y_test, y_pred) f1_score(y_test, y_pred, average='weighted') roc_auc_score(y_test, y_prob[:,1])
Regression
from sklearn.metrics import ( mean_absolute_error, mean_squared_error, r2_score, mean_absolute_percentage_error) import numpy as np mae = mean_absolute_error(y_test, y_pred) rmse = np.sqrt(mean_squared_error(y_test, y_pred)) r2 = r2_score(y_test, y_pred) mape = mean_absolute_percentage_error(y_test, y_pred)
Calibration & Threshold Tuning
AdvancedBrier Score
BS = (1/n) Β· Ξ£ (Ε·α΅’ β yα΅’)Β²
Measures probability calibration Β· 0 = perfect Β· 0.25 = random Β· lower is better
Threshold Tuning β Find Optimal
from sklearn.metrics import precision_recall_curve prec, rec, thresh = precision_recall_curve( y_test, y_prob) # Pick threshold maximising F1 f1s = 2*prec*rec/(prec+rec+1e-8) best_thresh = thresh[np.argmax(f1s)] y_pred_new = (y_prob >= best_thresh).astype(int)
Calibration
from sklearn.calibration import CalibratedClassifierCV # Isotonic or Platt scaling cal_model = CalibratedClassifierCV( model, method='isotonic', cv=5) cal_model.fit(X_train, y_train)
When to calibrate: Random Forest and SVM probabilities are often miscalibrated. Always calibrate if you're using probabilities for business decisions.
Model Evaluation Mastery Checklist
Self-AssessmentConfusion Matrix & Core Metrics
Draw and label all four cells of a confusion matrix (TP, FP, FN, TN)
Calculate Accuracy, Precision, Recall, and F1 from raw counts
Explain Type I (FP) and Type II (FN) errors with real examples
State when accuracy is a misleading metric and what to use instead
Explain the harmonic mean and why F1 uses it instead of arithmetic mean
PrecisionβRecall Tradeoff
Explain how adjusting the decision threshold affects precision and recall
Choose between precision and recall given a business problem
Use FΞ² score to weight recall or precision appropriately
AUC-ROC & Multi-class
Explain what the ROC curve plots (TPR vs FPR) and what AUC means
Interpret AUC values and state when AUC = 0.5 means random
Explain why PR-AUC is preferred over AUC-ROC for imbalanced data
Compute AUC-ROC and PR-AUC using sklearn
Apply macro, micro, and weighted averaging for multi-class F1
Interpret a sklearn classification report for all classes
Regression Metrics
Calculate MAE, MSE, RMSE, and RΒ² from predictions
Choose MAE vs RMSE based on outlier sensitivity requirements
Interpret RΒ² = 0 and RΒ² < 0 correctly
Cross-Validation
Implement stratified k-fold CV and report mean Β± std
Choose the right CV strategy for temporal and grouped data
Avoid data leakage in CV pipelines (scale inside CV loop)
Learning Curves & Calibration
Plot a learning curve and diagnose high bias vs high variance
Plot a validation curve to find the optimal hyperparameter range
Tune decision threshold to optimise F1 or recall as required
Explain probability calibration and when to apply it
Compute Brier Score and interpret it as a calibration measure
Select the right metric given task, class balance, and business cost