10.11 Chapter Review
10.11.1 Chapter summary
Defensible modelling begins with the decision, outcome, unit of analysis, and error consequences rather than with a preferred algorithm. Candidate methods should be compared with relevant baselines using validation that matches intended use, then examined for stability, subgroup performance, leakage, interpretability, and responsible use.
10.11.2 Common mistakes
- choosing a method before defining the outcome and decision use;
- evaluating performance on data used to fit or tune the model;
- allowing future or post-outcome information to leak into predictors;
- reporting one aggregate metric without examining error types or subgroups; and
- presenting association or predictive importance as proof of causation.
10.11.4 Exercises
Exercise 1. NVRW’s retention dataset contains 85 percent renewals. A model that predicts renewal for every member has 85 percent accuracy. Has the model succeeded?
Check Your Work
No. The model only reproduces the majority-class rate and identifies no non-renewals. Treat it as a baseline. Evaluation should include recall and precision for the actionable class, calibration, lift in the group NVRW could contact, and expected value under intervention capacity and cost.
Exercise 2. Use data/nvrw/derived/member_model.csv to define the target, prediction time, eligible population, features, and excluded information for a first-period retention model.
One Possible Answer
The target is first-period renewal among members with complete follow-up. The prediction time should be fixed before the renewal decision, such as after a defined early-engagement window. Eligible cases exclude censored members. Features may include information available by the prediction time, such as membership type, facility, acquisition channel, and early visits. Exclude later visits, renewal status fields, and any value calculated after the prediction time.
Exercise 3. Construct a baseline and compare logistic regression with one tree-based method. Use a split that prevents the same member from appearing in both training and test data.
Check Your Work
Use a majority-class or simple rate baseline, then split by member identifier so all records for one member remain in one partition. Fit preprocessing and tuning only on training data, compare both methods with decision-relevant metrics, and evaluate the selected method once on the untouched test set. Report uncertainty and whether the improvement over baseline is practically meaningful.
Exercise 4. Assume NVRW can contact only 150 members. Evaluate precision, recall, calibration, and expected contribution margin among the 150 highest-risk eligible members. Explain which metric best fits the decision.
Check Your Work
Rank only eligible members using predictions generated without test-set leakage. Precision among the top 150 indicates how concentrated non-renewals are in the contact group, recall shows how many total non-renewals the limited program reaches, calibration checks whether predicted risks are credible, and expected contribution margin connects targeting to value. The primary measure should reflect the actual objective and the cost and expected effect of contact, not accuracy alone.