10.8 Evaluation must match the decision

10.8.1 Regression metrics

• MAE describes the average absolute error in the outcome’s units.
• RMSE gives greater weight to large errors.
• MAPE expresses error as a percentage but behaves poorly near zero.
• R-squared describes explained variation under a given dataset and does not directly measure practical error.

10.8.2 Classification metrics

• Recall measures the proportion of actual positive cases identified.
• Precision measures the proportion of positive predictions that are correct.
• Specificity measures the proportion of negative cases correctly rejected.
• Calibration assesses whether predicted probabilities agree with observed frequencies.
• Lift compares the concentration of outcomes in a targeted group with the overall rate.

Metric choice should reflect action. If NVRW can contact only 500 members, precision and expected value among the 500 highest-risk eligible members may matter more than overall accuracy.