10.1 Begin with the purpose

Method selection begins with the decision, not the algorithm. Before modelling, state:

• the outcome or quantity of interest;
• the unit of analysis;
• the population and time period;
• whether the goal is description, explanation, prediction, classification, or segmentation;
• when the result would be used;
• what information would be available at that time;
• how performance will be evaluated; and
• what limitation would make the result unsuitable for use.

A model can perform well on a metric while failing the practical purpose. A retention classifier that identifies likely non-renewals only after the membership has expired is too late to support an intervention. A program-demand model may have acceptable average error but fail badly for the locations where staffing decisions are most costly.