Forecasting Evaluation Workbook & Backtesting Examples

{ "Title": "Forecasting Evaluation Workbook & Backtesting Examples", "IntroductionHtml": "

Welcome

This workbook helps you evaluate forecasting approaches so forecasts actually improve planning, staffing, inventory, and risk-aware decisions. Use it to run simple backtests, compare error metrics, capture diagnostics, and record the decision rules that convert probabilistic forecasts into action.

What you'll get

• Guided backtest parameter capture (rolling windows, holdout, expanding)
• Choice of error metrics (MAE, RMSE, MAPE, quantile loss)
• Fields for pasting a small example dataset to exercise the templates
• Spaces to record diagnostics, interpretation, and operational decision rules
• A saved record you can reuse, extend, or export

Quick start

1. Name the dataset and select frequency.
2. Choose a backtest method and window sizes.
3. Select the metrics you want to compute.
4. Paste a small CSV sample or describe the dataset structure.
5. Save the backtest record and use the notes to decide model replacement or operational thresholds.

Note: This workbook captures the backtest configuration and contextual notes. The platform can store your submission; advanced server-side scoring, automated charting, or direct connections to source data are described in the capability notes below.

", "SubmitLabel": "Save Backtest", "SuccessMessage": "Backtest record saved. You can run the workbook again, export results, or use this record to request an automated run where available.", "DataType": "forecasting-backtest-submission", "SchemaVersion": "1.0", "Fields": [ { "Key": "datasetName", "FieldType": "text", "Label": "Dataset name", "HelpText": "Short identifier for this dataset (e.g., 'Store A daily demand' or 'Revenue - Product line X').", "Required": true }, { "Key": "frequency", "FieldType": "select", "Label": "Data frequency", "HelpText": "Observation frequency for the series. This helps interpret window sizes and lead times.", "Required": true, "Options": [ { "Value": "daily", "Label": "Daily" }, { "Value": "weekly", "Label": "Weekly" }, { "Value": "monthly", "Label": "Monthly" } ] }, { "Key": "backtestMethod", "FieldType": "radio", "Label": "Backtest method", "HelpText": "Choose a backtesting approach suited to your cadence and data volume.", "Required": true, "Options": [ { "Value": "rolling", "Label": "Rolling window (walk‑forward) — recommended for evolving models" }, { "Value": "holdout", "Label": "Fixed holdout period — clear separation for evaluation" }, { "Value": "expanding", "Label": "Expanding window — useful when you want to use more history over time" } ] }, { "Key": "rollingWindowSize", "FieldType": "number", "Label": "Rolling window size (periods)", "HelpText": "Number of past periods used to train each model in a rolling test. Leave blank for holdout-only approaches.", "Required": false }, { "Key": "holdoutSize", "FieldType": "number", "Label": "Holdout size (periods)", "HelpText": "Number of final periods reserved for evaluation when using a fixed holdout. Choose at least as long as the forecast horizon to avoid leakage.", "Required": false }, { "Key": "errorMetrics", "FieldType": "checkbox", "Label": "Error metrics to compute", "HelpText": "Select metrics to compute when you run or export this backtest. Each gives a different view — use more than one when practical.", "Required": true, "Options": [ { "Value": "mae", "Label": "MAE — Mean Absolute Error (robust, interpretable in units)" }, { "Value": "rmse", "Label": "RMSE — Root Mean Square Error (penalizes large errors)" }, { "Value": "mape", "Label": "MAPE — Mean Absolute % Error (use cautiously with small actuals)" }, { "Value": "quantile", "Label": "Quantile loss — for probabilistic forecasts or quantiles" } ] }, { "Key": "includeProbabilistic", "FieldType": "yesno", "Label": "Include probabilistic forecasts", "HelpText": "Select yes if your forecasts include quantiles or prediction intervals (e.g., 10th, 50th, 90th percentiles). This alters the diagnostics we recommend.", "Required": true }, { "Key": "sampleDataCsv", "FieldType": "textarea", "Label": "Paste sample data (CSV)", "HelpText": "Paste a small CSV sample to exercise the templates and capture structure. For simple point forecasts use header: date,actual,forecast. For probabilistic forecasts add columns like forecast_q0.1, forecast_q0.5, forecast_q0.9. One row per date.", "Required": false }, { "Key": "diagnosticsToRun", "FieldType": "checkbox", "Label": "Visual diagnostics & decomposition", "HelpText": "Select the diagnostics you'd like to note or run when automations are available.", "Required": false, "Options": [ { "Value": "ts_decomposition", "Label": "Trend/seasonal decomposition" }, { "Value": "residual_autocorr", "Label": "Residual autocorrelation (ACF/PACF)" }, { "Value": "error_distribution", "Label": "Error distribution / bias check" }, { "Value": "calibration", "Label": "Probabilistic calibration (PIT, quantile coverage)" } ] }, { "Key": "decisionRules", "FieldType": "textarea", "Label": "Decision rules & action mapping", "HelpText": "Describe how forecast outcomes should translate into operational actions. Be concrete: include thresholds, lead times, safety stock rules, staff scheduling triggers, or escalation steps (e.g., 'If P(demand>stock) > 0.85 for horizon 14 days, create expedite order').", "Required": false }, { "Key": "interpretationNotes", "FieldType": "textarea", "Label": "Notes and interpretation", "HelpText": "Summarize diagnostic findings, model comparisons, and recommended next steps. Capture confidence, known data issues, and suggested experiments (e.g., seasonal features, promotional effects).", "Required": false } ] }