Forecasting Backtest & Evaluation Workbook

Welcome

This workbook helps teams make forecasting more reliable by turning backtesting and evaluation into a clear, repeatable operational practice. Use it while you plan a forecast release, validate a new model, or audit existing forecasting processes. It focuses on practical choices you can implement with operational data and then translate into planning, staffing, inventory, or risk-aware actions.

Who this is for

Owners, planners, analysts, and managers who need forecasts to inform concrete decisions (staffing, purchasing, production, budgets). No advanced statistics required—this workbook explains essential terms and gives concrete templates you can adapt.

What you'll get

• Backtest design checklist and rolling-origin recipe
• Guidance for selecting and interpreting error metrics (MAPE, RMSE, MAE, quantile loss, CRPS)
• Probabilistic forecast checks: calibration and sharpness
• Templates for recording results and translating them into operational triggers
• Common pitfalls and diagnostics to avoid misleading conclusions

How to use this workbook

1. Complete the Backtest Plan Template (define horizon, granularity, holdout strategy).
2. Run the backtest using your model(s) and capture results in the Results Template.
3. Compute selected error metrics and carry out probabilistic checks if applicable.
4. Discuss operational implications using the Trigger & Actions template and agree next steps.

Step 1 — Backtest Plan Template

Before running anything, record these decisions so the exercise is reproducible.

• Forecast purpose: (e.g., weekly demand for SKU X to set production).
• Forecast horizon(s): (e.g., 1 week, 4 weeks, 12 weeks).
• Granularity: (daily, weekly, SKU-level, product family).
• Origin cadence: (how often forecast is produced—weekly, monthly).
• Backtest window: start and end dates for historical tests.
• Holdout design: rolling-origin (preferred) or fixed holdout — specify fold size and number of folds.
• Data preprocessing rules: treatment of outliers, imputation, special events and promotions.
• Evaluation owners: who calculates metrics, who reviews results, who approves triggers.

Quick guidance: prefer a rolling-origin backtest for operational models because it better mirrors running forecasts over time and exposes temporal model degradation.

Step 2 — Choosing error metrics

Pick metrics that match the decision you support. Record why each metric was chosen and what an acceptable range is for your use case.

Common metrics (what they tell you)

• MAE (Mean Absolute Error): average absolute error in units — easy to interpret and robust to outliers.
• RMSE (Root Mean Squared Error): penalizes large errors — useful when big misses are especially costly.
• MAPE (Mean Absolute Percentage Error): error relative to actuals — simple but unstable when actuals are near zero.
• Symmetric MAPE / sMAPE: mitigates some MAPE bias but has its own tradeoffs.
• Quantile loss (pinball loss): for probabilistic forecasts or when asymmetric costs exist; evaluate for each quantile of interest.
• CRPS (Continuous Ranked Probability Score): a single score for full predictive distributions; lower is better and comparable across models producing distributions.

Rules of thumb

• Use at least one scale-dependent metric (MAE or RMSE) and one scale-independent metric (MAPE or percentage-based) when comparing series of different sizes.
• If your decisions are asymmetric (e.g., under-forecast is worse than over-forecast), include asymmetric loss or quantile-level metrics.
• Avoid relying solely on MAPE for series that contain zeros or small values.

Step 3 — Rolling-origin backtest (recipe)

A rolling-origin backtest simulates how forecasts would have been produced over time.

1. Choose initial training window ending at time t0.
2. Train model on data up to t0 and generate forecasts for horizon H (e.g., 4 weeks ahead).
3. Record forecast vs actual for that origin.
4. Advance origin by one cadence step (e.g., one week), re-train using data up to t0+1, and repeat until you reach the backtest end date.
5. Aggregate errors by horizon and by origin to understand time-varying performance.

Notes:

• If model retraining is expensive, use a defined retrain cadence (monthly) that matches production practice — but ensure the backtest mimics how the model will actually be maintained.
• Record the sample size (number of origin-horizon pairs) used to compute each metric to understand statistical stability.

Step 4 — Probabilistic forecasts: calibration & sharpness

If you produce prediction intervals or full distributions, evaluate two dimensions:

• Calibration (reliability): does the nominal coverage match observed coverage? Example: for 90% prediction intervals, ~90% of actuals should fall inside the interval. Use coverage checks and PIT (probability integral transform) histograms.
• Sharpness: how concentrated are your predictive distributions? Sharpness is desirable only when calibration is acceptable — overly sharp, miscalibrated predictions are dangerous.

Practical checks:

• Compute observed coverage at several levels (50%, 80%, 90%) and compare to nominal.
• Plot PIT histograms — uniform PIT indicates good calibration; U-shaped indicates underdispersion; mound-shaped indicates overdispersion.
• Compute CRPS to compare full-distribution performance across models.

Step 5 — Results Template (what to capture)

For each model and horizon, capture:

• Model name / version
• Origin date
• Horizon (e.g., 1 wk, 4 wk)
• MAE, RMSE, MAPE (or chosen metrics)
• Coverage at chosen intervals (if probabilistic)
• CRPS or quantile losses (if applicable)
• Number of origin-horizon pairs used
• Notes about special events or data issues for that origin

Example (single line): Model-A | 2023-01-02 origin | 4-week horizon | MAE=120 | RMSE=190 | MAPE=6.5% | 90% coverage=88% | CRPS=150 | n=52

Step 6 — Translate evaluation into operational triggers

Evaluation is only useful if it leads to decisions. Use the following template to co-design triggers with operations.

• Metric monitored: e.g., 4-week MAE at SKU family level.
• Baseline / acceptable range: e.g., MAE <= 150 units or MAPE <= 8%.
• Trigger condition: metric breaches acceptable range for 2 consecutive origins.
• Action when triggered: (a) Investigate recent promotions/returns, (b) escalate to model owner, (c) switch to fallback forecasting method, (d) adjust safety stock.
• Responsible roles: analyst, planner, ops manager
• Timeline for remediation: e.g., 48 hours to triage, 5 business days to deploy short-term fix.
• Post-mortem: capture root cause and preventive actions.

Common diagnostics & next checks

• Data leakage check: ensure no future information leaked into training features.
• Seasonality & calendar effects: check whether errors increase around holidays or promotions — consider separate models or features for these periods.
• Horizon sensitivity: compare metric by horizon — some models are accurate short-term but degrade rapidly.
• Segment stability: examine performance across SKUs, regions, or channels — poor aggregate metrics can hide problematic segments.
• Bias checks: compute mean forecast error (MFE) to detect systematic over- or under-forecasting.

Practical exercises (apply to your data)

1. Pick a representative SKU or product family. Run a 12-month rolling-origin backtest at weekly cadence and record metrics by horizon.
2. Plot MAE or MAPE over origins to see if performance drifts. If drift exists, annotate potential causes (data changes, new promotion types).
3. If you use probabilistic forecasts, compute 50%, 80%, 90% coverage and make a PIT histogram. Decide whether recalibration is needed.
4. Define one trigger based on a metric and run a simulated monitoring window to see how often it would have fired historically.

Checklist before declaring a model 'production-ready'

• Backtest mirrors production cadence and retraining practice.
• Enough origin-horizon samples for stable metrics (n documented).
• Key segments (top SKUs/locations) tested individually.
• Probabilistic forecasts calibrated and acceptably sharp (if used).
• Operational triggers and owners agreed and documented.
• Monitoring plan and post-trigger actions defined.

Common pitfalls and how to avoid them

• Overconfident single-point forecasts: Always complement point forecasts with error estimates or prediction intervals when decisions depend on uncertainty.
• Misleading metric choice: Pick metrics that align with cost structure; do not rely solely on MAPE if series have frequent zeros.
• Insufficient backtest horizon: Short backtests may miss seasonality and regime changes—use multiple years when available.
• Ignoring operational constraints: Ensure lead times, capacity, and inventory rules are considered when translating forecasts into actions.
• Undisclosed assumptions: Document data cutoffs, promotions, and holiday handling so reviewers can interpret results correctly.

Templates you can copy

Use these headings to build simple spreadsheets or records in your systems.

Backtest Plan

Fields: Forecast purpose | Horizon(s) | Granularity | Origin cadence | Backtest start/end | Holdout design | Data rules | Owners

Results row

Fields: Model | Origin | Horizon | Actual | Forecast | Error | MAE | RMSE | MAPE | Coverage@90% | CRPS | Notes

Trigger & Action

Fields: Metric monitored | Baseline | Trigger condition | Action(s) | Owner | SLA | Post-mortem notes

Reflection questions

• Which metric best aligns with our biggest operational cost of forecast error?
• How would our triggers behave during extreme but plausible events (promotion spikes, supply disruption)?
• What segments should be monitored separately because aggregate metrics hide important failure modes?

Next steps & recommended resources

• Automate rolling-origin backtests on a cadence matching production.
• Store backtest results and alerts so you can measure model drift over time.
• Explore recalibration methods when probabilistic forecasts are miscalibrated (e.g., isotonic regression, Platt scaling for quantiles).
• Suggested reading: articles on CRPS, quantile regression and operationalizing forecasts.

How this workbook may be adapted

Copy and adapt the templates to reflect your product taxonomy, lead times, and cost structure. Keep the backtest plan with each run so future reviewers can understand historical context.