Reproducible Compute Starter Kit — containers, workflows, and provenance

Make your compute runs repeatable, traceable, and affordable

Reproducible compute means someone else (or you in six months) can rerun an analysis and reach the same results with a clear audit trail of inputs, code, environment, parameters, and costs. This guide gives a compact, practical starter playbook: concrete container and workflow patterns, what provenance to capture, and cost-aware execution tips you can apply today.

Why this matters

Unreproducible runs waste time, block collaboration, and make results untrustworthy. A small upfront discipline—pinned dependencies, immutable artifacts, recorded provenance, and a lightweight release checklist—dramatically reduces friction for collaborators, reviewers, and downstream engineering.

Core components

1. Immutable execution artifacts — container images (with digests), environment lockfiles, or binary bundles so the runtime is portable and pinned.
2. Declarative workflows — workflow descriptions (Nextflow, CWL, WDL, Snakemake) that encode the DAG, parameters, and resource requests.
3. Provenance capture — explicit metadata about inputs, versions, config, execution, and checksums following simple standards (W3C PROV, RO-Crate, or a team convention).
4. Cost-aware execution — choose compute types, caching, preemption strategies, and partitioning to balance repeatability and budget.

Containers and pinned dependencies — practical notes

Why: containers capture OS-level dependencies; pinning versions prevents drift. How to do it well:

• Start from a small explicit base image (for example: python:3.10-slim) and avoid implicit "latest" tags.
• Commit a dependency lockfile (pip freeze, poetry.lock, requirements.txt with hashes, conda env.yml) into the repo.
• Publish images with immutable digests and record the digest in provenance (not just the tag).
• Include a minimal Dockerfile that defines the build and a reproducible build process (build args, multi-stage when appropriate).

Singularity / Apptainer note

For HPC, provide a Singularity recipe (definition file) that references the same base and pinned packages. Treat it as another immutable artifact and record its build hash.

Workflow engine patterns and reproducibility checkpoints

Use a workflow engine to encode steps, inputs, outputs, and resource needs. Example reproducibility checkpoints to include in each workflow run:

• Workflow descriptor version (git commit hash or release tag).
• Container image digest(s) for every task.
• Input manifest with filenames and checksums (SHA256).
• Config file used (parameters, resource settings) recorded with a checksum and path.
• Execution metadata: start/end timestamps, host/cluster id, workflow engine version, and run id.

Provenance capture: what to record

Keep provenance practical: capture what someone needs to rerun and verify outputs. Minimum recommended fields:

• Repository URL + commit hash or release tag.
• Container image name + digest(s).
• Input dataset references and checksums (SHA256), with a short description.
• Exact command lines / workflow invocation and parameter values.
• Random seeds and nondeterministic sources (e.g., library RNG settings).
• Workflow engine name and version, scheduler/cluster details where relevant.
• Output checksums and sample expected metrics to ease verification.
• Optional: SBOM or dependency list for deeper auditing.

Store provenance as a small machine-readable artifact alongside results (JSON or RO-Crate). Example formats to consider: W3C PROV, RO-Crate for packaging, or a simple JSON schema your team agrees on.

Cost-aware reproducible execution

Reproducibility shouldn't bankrupt you. Consider these patterns:

• Use spot/preemptible instances for non-interactive runs and add robust retry/ checkpointing in the workflow.
• Right-size resources in the workflow descriptor (CPU/memory) to avoid waste and ensure consistent runtimes.
• Cache intermediate artifacts to reuse earlier results and avoid rerunning expensive steps.
• Partition large jobs into smaller deterministic chunks that can be rerun selectively.
• Record cost metrics per run (cloud billing tags, runtime * instance type) to inform budgeting and future optimizations.

Minimal reproducible-release checklist

Before publishing or handing off a run, make sure you have:

• Code repository URL and commit hash (or release tag).
• Container image name(s) and digest(s).
• Workflow descriptor and exact invocation command (with config file).
• Input manifest with checksums and a short data-use note.
• Output checksums and a short verification test (one or two commands to validate outputs).
• Recorded random seeds and any nondeterminism notes.
• Estimated cost for the run and resource choices used.
• Provenance JSON or RO-Crate bundle attached to the results directory or DOI if published.

Getting started: five practical steps

1. Pick a single representative pipeline and make it reproducible end-to-end: pin dependencies, build and push a container, and record the image digest.
2. Add an input manifest and an outputs verification test so someone can check a rerun quickly.
3. Instrument the workflow to emit a small provenance JSON at the end of each run (commit, image digests, inputs + checksums, config checksum, run id).
4. Run with a controlled small dataset and verify the verification test passes; record runtime and cost.
5. Publish the bundle (repo tag, container digest, provenance file) and run the checklist. Iterate on any gaps you find when rerunning later.

Where to go next

When your team is comfortable with the starter pattern, consider these next steps:

• Automate provenance capture with tools like ReproZip, RO-Crate producers, or built-in workflow provenance plugins.
• Create a reproducibility CI job that builds containers, runs a small test dataset, and verifies outputs on each PR.
• Package this starter guide into a team toolkit (templates, example Dockerfile, example workflow, and an interactive checklist) so projects can copy and adopt easily.

Quick reference and resources

• W3C PROV: provenance data model.
• RO-Crate: packaging research outputs and metadata.
• ReproZip: recording execution traces for reproducibility.
• Workflow engines: Nextflow, CWL, WDL, Snakemake — pick what fits your team.

Good reproducibility is a combination of technology and small process changes. Start small, prove value with a single pipeline, and then generalize the patterns into templates and automation that your team can reuse.