Modeling & simulation playbook — reproducible computational experiments

Purpose and audience

This playbook helps researchers, engineers, data scientists, and simulation teams produce trustworthy, reproducible computational experiments. It focuses on four connected goals: clear problem scoping, model selection with verification, uncertainty quantification (UQ) and sensitivity analysis, reproducible compute and provenance capture, and validation plus transparent reporting.

Use this playbook as a project checklist, a template for team practices, or a starting point for building interactive checklists and reproducibility tooling.

Core principles

• Design for decision use: define what decision the model should support and what level of fidelity is required.
• Document assumptions up front: make assumptions explicit and test their impact.
• Separate verification, validation, and UQ: confirm the code is correct (verification); check the model against reality (validation); quantify uncertainties that matter (UQ).
• Capture reproducible provenance: inputs, code versions, environment, random seeds, and run manifests that let others reproduce results.
• Report transparently: include limitations, sensitivity, and uncertainty summaries—don’t hide them.

1. Problem & scope definition

Begin by writing a short, machine- and human-readable project brief that answers:

• What question or decision is this model intended to support?
• Who are the stakeholders and what actions might follow from the model?
• What are the required outputs, performance metrics, and acceptable error bounds?
• What data are available, and what data are assumed or synthetic?
• What timeline and compute budget apply?
• What regulatory, ethical, or safety constraints affect model use?

Capture this metadata for every experiment. A minimal metadata template should include:

• Project name, PI/owner, date, purpose
• Intended decision(s) and required fidelity
• Primary datasets (location, version, access)
• Model type and major assumptions
• Compute resources and run budget

2. Model selection, assumptions, and verification

Model selection should balance complexity, interpretability, and available data. Always record alternatives considered and why one approach was chosen.

Verification checklist (ensure the implementation does what the model specification says)

1. Code review and unit tests for core components (numerical solvers, boundary conditions, data parsers).
2. Regression tests that lock down outputs for fixed inputs.
3. Convergence and grid/refinement studies for numerical methods.
4. Comparison to analytical solutions or simple limiting cases where available.
5. Round-trip tests: small, known-input experiments that reproduce expected behavior.
6. Automated test suite integrated into CI so verification runs on change.

Document every assumption (boundary conditions, initial conditions, reduced physics, surrogate models) and tag it by likely impact (low / medium / high). Plan targeted verification or sensitivity tests for high-impact assumptions.

3. Uncertainty quantification & sensitivity analysis

Decide which uncertainties to address: parameter (aleatory), structural/model-form (epistemic), and input/data uncertainty. Design a UQ plan that ties uncertainty sources to decision-relevant metrics.

UQ plan elements

• Define target metrics (e.g., mean, quantiles, exceedance probability, decision thresholds).
• Choose propagation method: Monte Carlo, Latin Hypercube, Quasi-Monte Carlo, polynomial chaos, or surrogate-based sampling depending on cost and smoothness.
• Budget the number of runs and trade off fidelity vs. sample size (use surrogates when runs are costly).
• Plan global sensitivity analysis (Sobol indices, variance-based methods) to identify influential parameters.
• Design conditional analyses to explore model-form uncertainty: alternative model structures, competing physics, or different parameterizations.
• Decide validation checks for propagated uncertainty: coverage tests, calibration checks, posterior predictive checks if Bayesian.

Document working assumptions for UQ (independence, distributions, priors) and the rationale for chosen methods.

4. Compute reproducibility: capture the environment and run manifest

Reproducibility means others can re-run your experiment and (within specified tolerances) obtain the same results. Capture everything that affects outputs.

Key elements to record

• Code repository URL and commit hash (tag releases used for published runs).
• Container image (Docker/Singularity) or environment specification (Conda env, pip freeze), with image digest when possible.
• Exact input data files (paths and version identifiers) and a checksum/hash for each file.
• Run manifest: command-line, configuration file, parameter file, and random seeds.
• Hardware notes when results are hardware-dependent (GPU model, CPU architecture, thread count, BLAS linking).
• External service versions (e.g., MPI version, solver library versions).

Practical steps

1. Package the runtime in containers and publish or snapshot the image to a registry.
2. Use a workflow engine (Snakemake, Nextflow, CWL) or scripts that can be re-executed end-to-end.
3. Pin dependency versions and record hashes. Do not rely on unconstrained “latest” packages.
4. Store input snapshots (or a link to immutable archival storage) and record access methods.
5. Record random seeds and, when using parallel RNG, document strategy (e.g., reproducible RNG streams per process).
6. Automate run manifests: produce a machine-readable run.json containing the items above for every published experiment.

Example minimal manifest (publish alongside results):

{
  "project": "HeatTransferStudy",
  "commit": "abc123def",
  "container": "registry/org/image@sha256:...",