11.7 Time-ordered validation

Forecasting methods should be evaluated as they would be used. A random train-test split destroys the time order. Instead, fit on earlier periods and evaluate on later periods.

Rolling-origin evaluation repeats this process across several cutoffs:

1. fit using data available at the first cutoff;
2. forecast the chosen horizon;
3. record error when actual values become available;
4. move the cutoff forward; and
5. repeat.

Compare MAE, RMSE, scaled error, interval coverage, and performance at the required horizon. A method that is strong one month ahead may be poor twelve months ahead.

Worked Example: Forecasting Monthly Visits

NVRW compares a seasonal naive baseline, a three-month moving average, seasonal regression, ETS, and seasonal ARIMA for monthly facility visits. Evaluation uses rolling cutoffs with a twelve-month horizon.

ETS has the lowest average MAE, but its errors are concentrated at Lakeside after a facility expansion. The team reports separate location models and notes that the post-expansion period provides limited evidence for long-range forecasting.